Summary

Neural networks are widely assumed to use superposition to represent more features than they have dimensions. A stronger claim is that they also compute in superposition (CiS), i.e., implement more nonlinear functions than they have neurons (Hänni et al. 2024). CiS remains poorly understood, and until recently there were no examples of it arising through training rather than being hand-designed. Toy models of CiS are desirable for two reasons: they're small enough that we can hope to fully reverse-engineer them, and they provide ground-truth solutions against which to test interpretability methods, in particular parameter decomposition methods like APD and SPD, which is what the compressed computation toy model of Braun et al. 2025 was originally built for. That model is a single hidden layer of 50 ReLU neurons that must compute the ReLU of each of 100 sparse input features. But Bhagat et al. 2025 then showed that the trained solution wasn't actually computing in superposition: its advantage over naive baselines came from a mixing term introduced by the residual connection, not from sharing neurons across features.

In this work, we train a version of this model that does perform computation in superposition. Following Bhagat et al.'s analysis we drop the residual connection, and, the key change, we train under loss instead of the usual (the idea of changing the loss came from a comment by Lucius Bushnaq). The intuition is that punishes outlier errors more heavily, so it favors superposition solutions that handle all features roughly equally well over the naive solution finds, which computes half the features well and ignores the rest. Nothing appears to be special about exponent 4 — every exponent above 2 that we tried results in an identical superposition solution.

We then reverse-engineer the solution: the encoder implements redundancy by assigning each feature a sparse binary codeword over 5–7 neurons, and the decoder is close to a scaled pseudoinverse of the encoder — each feature is read out along the same neurons that encode it. Given the codewords, a description with just three scalars (on-code value, off-code value, decoder scale) recovers most of the trained network's performance (reaching its loss), and we validate the picture by building equivalent networks from hand-designed binary codes, which perform comparably.

Key results

1. We train a toy model of compressed computation with loss and argue that it performs computation in superposition.
2. We reverse-engineer the trained network: it assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a 3-scalar parameterization (on-code value, off-code value, decoder scale) recovers its loss.
3. We validate this description by substituting hand-designed binary codes in the same 3-parameter family, yielding equivalent networks at the trained model's loss.

Why we think this matters

First, this gives one of the first toy models of computation in superposition arising through training rather than being designed by hand (concurrently with Gibson 2026, who studies a related but different task).

Second, it restores the compressed computation setup to its original purpose. It was built as a testbed for parameter decomposition methods, then shown not to contain the phenomenon it was meant to test them on. Now it does — and since we've reverse-engineered the solution, it comes with a known ground-truth mechanism to compare decompositions against, which is what we plan to use it for next.

The rest of this post is copied from the paper, except for the abstract and the list of contributions. Code is available at github.com/FranciscoHS/toy-model-cis-code.

Introduction

Neural networks are widely assumed to make use of superposition to represent more concepts than they have dimensions (Elhage et al., 2022). A natural follow-up is whether they also compute in superposition, i.e., implement more non-linear functions than they have dimensions (Hänni et al., 2024; Adler & Shavit, 2024).

While representation in superposition is increasingly well understood, computation in superposition (CiS) is not. In particular, until very recently (Gibson, 2026) there were no toy models of computation in superposition in the wild, i.e., arising through training rather than being hand designed. Such toy models are desirable: they allow studying CiS at a scale where we can hope to fully reverse engineer the trained networks.

Braun et al. (2025) introduced the toy model of compressed computation, where a single-hidden-layer network must compute more ReLUs than its hidden width. However, Bhagat et al. (2025) showed that this toy model did not actually elicit computation in superposition.

In this work, we train a version of the compressed computation toy model under loss rather than and argue that the resulting network performs computation in superposition.

Related Work

Comparable empirical work. Gibson (2026) is concurrent work similar in spirit to ours: both partially reverse-engineer one-layer ReLU networks that learn binary codes, and the aspects that remain unexplained are nearly identical — non-uniform on-codeword values, a decoder that deviates from a scaled pseudoinverse, and a trained network that outperforms designed alternatives. The key difference is the task. Both networks take continuous sparse inputs, but ours has a regression task (compute the ReLU of each input), whereas Gibson's network must identify which inputs are active, outputting a fixed value at each active index regardless of its magnitude.

Theoretical constructions for CiS. Gibson (2025) considers a task in which a continuous input passes through one of many small circuits sharing a single wider network, with exactly one circuit active per forward pass. They provide a hand-designed zero-error construction for computation in superposition via sparse memory blocks; we find a sparse-coding solution arising through training.

Random sparse binary codes. Hänni et al. (2024) proposes random sparse binary codes for computation in superposition. Our work appears to be an empirical instance of their constructions.

The compressed computation toy model. Braun et al. (2025) introduced the toy model of compressed computation, and Bhagat et al. (2025) showed that it does not elicit computation in superposition. We train a version of this model under loss and show that it does.

Methodology

Our network (Figure 1) is the compressed-computation toy model of Braun et al. (2025) with the residual connection removed (see below). The input is -dimensional, with each feature being zero with probability and drawn uniformly from with probability ; we set , giving two active features in expectation. A single hidden layer of ReLU neurons without bias acts directly on the input through trainable weights and , and the task is to compute the elementwise ReLU of the input. Braun et al. (2025) additionally embed the input as random near-orthogonal directions in a higher-dimensional () space through a fixed random matrix with unit-norm rows, read back out by the transpose , so that the features are not axis-aligned — the more realistic and challenging setting for the parameter-decomposition methods the model was built to test. That embedding turns out to be incidental to the computation-in-superposition solution: training with it recovers an identical solution. We therefore present the simpler axis-aligned model in the main text and defer the embedded variant to Appendix A.

Figure 1. Network architecture: a linear encoder , 50 ReLU neurons without bias, and a linear decoder . The input and output are 100-dimensional; the task is to compute the elementwise ReLU of the input.

Beyond setting aside the incidental embedding, we also remove the residual connection, motivated by Bhagat et al. (2025)'s analysis. They show that, in Braun et al.'s model, the residual connection is equivalent to a mixing matrix in the target, , where couples input to target for — and that this , not superposition, is what lets the model beat the naive baseline of ignoring half the features. Removing the residual connection sets , so the target is exactly and any substantial improvement over the naive baseline must come from superposition. This removal is necessary but not sufficient: Bhagat et al. (2025) showed that dropping the residual alone does not elicit computation in superposition, and we confirm it — trained under the usual loss our model still finds the naive solution (see the next section).

The change we make that elicits CiS is the loss. We train under loss, the average fourth power of the elementwise error, rather than the usual . We train with Adam (learning rate , cosine annealing schedule), batch size , for steps; and are initialized entrywise from and . In the next section we also sweep the loss exponent, including standard . The intuition behind making this change is that penalizes outlier errors more than , and hence should prioritize networks approximating all features roughly equally well (superposition) rather than computing the output for some features perfectly and for others not at all.

Results

Evidence for computation in superposition

As evidence that training elicits superposition, we compare the per-feature loss of the - and -trained networks (Figure 2). Training under results in a "naive solution" that learns half the features perfectly and ignores the rest (red). Under , the trained network instead learns all 100 features approximately equally well. We further compare against another no-superposition baseline, namely one that represents half the features exactly and emulates a bias on the rest by adding a constant offset to their outputs (following Bhagat et al. (2025)). At our sparsity the offset helps only marginally, with the emulate-bias baseline having loss lower than the naive baseline, still higher than the solution.

Figure 2. Per-feature MSE on samples: the strongest non-superposition baseline and the trained networks at several loss exponents. Each violin pools the per-feature MSE (over the features, and over seeds for the trained networks); black dots are medians. The emulate-bias baseline and (reds) are bimodal — half the features handled, half left near the do-nothing baseline of (dashed), which the bias offset lowers only marginally. Every exponent above (blue) instead concentrates the error into a narrow band far below the baselines, spread evenly across all features.

As there are more features than neurons, achieving low error on all features requires sharing neurons across features, i.e., superposition. Eliciting this solution is not unique to , as Figure 2 shows: every exponent above produces the same concentrated distribution. To quantify this, we summarize each network by the coefficient of variation of its per-feature MSE across the 100 features — the standard deviation divided by the mean. We use the coefficient of variation rather than the raw standard deviation because the mean per-feature loss varies greatly across exponents, so a scale-free measure is needed to compare them; it is small when the error is spread evenly across features (indication of superposition) and large when a few features carry most of the error (the naive solution). Every exponent above that we tried () gives a superposition-like solution, with coefficients of variation between and ; instead gives the naive solution, with a coefficient of variation of . Across seeds the coefficient of variation varies by at most throughout. is thus a representative choice rather than a special one; Figure 2 also shows and , which cluster with well below the baselines.

The mechanism

We characterize the -trained network's mechanism through three observations about its encoder , which maps each input feature to the neuron pre-activations it induces, and its decoder .

Figure 3 shows the distribution of the entries of the encoder , pooled over all feature columns. The entries fall into two well-separated groups (values reported as mean standard deviation): a large mass of small negative values () and a smaller mass of large positive values (). We call the support of the large entries the codeword of feature , written — the subset of neurons that fire when feature is active, with codeword length . Stacking these gives a binary matrix whose pattern matches the shape of : large positives where , small negatives where .

The codewords are quite regular (Figure 4): every feature uses - neurons (mean ), and every neuron participates in - codewords (mean ). The code is thus close to biregular — every codeword nearly the same length, and every neuron shared by nearly the same number of codewords — so the decoding load is spread near-equally across neurons.

Figure 3. Encoder entries are bimodal: pooled histogram of all entries of the encoder over the feature columns. Bars are binned in steps of with edges aligned to zero; bar height is the number of entries in each bin, with those entries scattered over the bar at their true value (vertical position is jitter, for visibility). The entries split into a large off-code cluster of small negatives () and a smaller on-code cluster of large positives ().

Figure 4. The binary code is regular: number of large encoder entries (those exceeding , i.e. on-code entries) per feature column (blue, left axis) and per neuron row (red, right axis). Every feature uses - neurons (mean ) and every neuron is shared by - codewords (mean ); the tight bimodal distribution shows the code is close to biregular.

If the codewords serve as the network's internal identifier for each feature, then transplanting feature 's hidden activation values onto feature 's codeword neurons should make the decoder output feature . We verify this directly: for every pair of features of equal codeword length we transplant feature 's hidden magnitudes onto feature 's codeword neurons, and the decoder's top output is in of cases (Figure 5).

Figure 5. Codeword swap test: the hidden magnitudes produced by feature , placed on feature 's codeword neurons, decode to . The codeword positions, not the magnitudes, determine the decoded feature.

Finally, the decoder seems to do little more than invert the encoder: the decoder has Frobenius cosine with a scaled pseudoinverse of the encoder (the Frobenius cosine is the cosine similarity between the two flattened matrices). To confirm, we replace the trained decoder outright with a scaled pseudoinverse of the encoder and find the loss increases only slightly ( the trained network), so the decoder is well described as a pseudoinverse read-out of the codewords. Because the codewords have small pairwise overlap, this pseudoinverse is itself close to a scaled transpose of the encoder, so the decoder ends up looking much like the encoder: each feature is read out along essentially the same neurons that encode it.

Together, these findings paint the picture in Figure 6: a binary-code encoder followed by a pseudoinverse decoder. One feature of this picture is that the decoded output is attenuated: the network systematically under-shoots the true ReLU. This is a response to interference — because codewords overlap, an active feature leaks through its shared neurons into the outputs of the (far more numerous) inactive features, and shrinking every active output keeps that leakage small. The single-feature input–output response in Figure 12 (Appendix B) shows this directly.

Figure 6. Mechanism for a single active feature: the encoder produces large positives on the feature's codeword neurons and small negatives elsewhere; ReLU zeros the negatives; the pseudoinverse decoder reads the codeword and outputs a peak at the same index.

A 3-parameter description given a binary code

The mechanism depicted in Figure 6 suggests a simple ansatz. We extract the binary codeword matrix from the trained network by classifying each encoder entry as on- or off-code, then parameterize a new encoder with one value on the support and one off it, , and set the decoder to a scaled pseudoinverse of it, . This leaves three free scalars: the on-code value , the off-code value , and the decoder scale .

Fitting these three scalars to the trained network's codewords gives an loss that of the trained network, far below the non-superposition baselines (naive , emulate-bias , random network ).

The support carries little information. We test how much the specific support matters by replacing with a randomly generated codeword matrix of a given codeword length . We consider two families: biregular codes (every codeword the same length, every neuron shared by the same number of codewords) and random codes. We try each family both as generated and after applying random edge swaps that reduce pairwise codeword overlaps (Appendix C), expecting biregularity to balance load across neurons and uniform overlaps to minimize the worst-case interference that penalizes most.

Figure 7. loss of 3-parameter ansatz networks: Loss of networks built from synthetic binary codes (biregular and random) as a function of codeword length , normalized by the trained model's loss. Solid lines apply overlap-minimizing edge swaps (five seeds each; error bars are standard deviations, scatter points are individual seeds); dashed lines are the same codes without edge swaps. Biregular with edge swaps is best at the trained loss; without edge swaps both families stay within . No synthetic code beats the trained network.

Figure 7 shows the result. Overlap minimization matters: without it both families sit at - the trained loss, while with it the best designed code — biregular — reaches , marginally beating the three-scalar version of the trained network's own code. Up to the choice of code family and codeword length , then, the network's performance is captured by just three scalars, and the precise support carries little extra information.

Open Questions

The three-scalar ansatz built from the trained codewords reaches the trained loss, and the best hand-designed code (Figure 7) comes no closer. Since the ansatz already uses the trained network's own support — so the codewords, including their lengths, match — what separates it from the trained network is just the encoder's specific per-entry values and the decoder. Table 1 decomposes how much each accounts for: relaxing the encoder or the decoder closes part of the gap, and the network reaches only when both are free. We understand each piece only partially.

Table 1. What each ingredient accounts for. For each construction we list the encoder, the decoder, and the resulting loss relative to the trained network (). "Trained support": the binary codeword pattern (which neurons each feature fires), read off the trained encoder. "3 scalars": the on-code/off-code/decoder-scale ansatz . "Free per-entry": every encoder entry fit freely. "pinv": decoder fixed to a scaled pseudoinverse of the encoder. Designed codes use codeword length ; "overlap-min." applies the edge swaps of Appendix C (Figure 7). The network reaches only when both encoder and decoder are free.

First, the per-entry encoder values. Replacing the two on/off scalars with the trained network's individual entries, while still decoding with a scaled pseudoinverse, lowers the loss to ; but we do not fully understand what these values encode. On-codeword values are not uniform, and enforcing uniformity harms performance. We observe that neurons used in more codewords have lower on-codeword values (), as one would expect if the network down-weights overloaded neurons; this accounts for of the variance in codeword values, so it is most but not all of the story. Notably, the values still vary even in a designed biregular code, where every neuron has the same degree: with this load-balancing effect removed entirely, the fitted per-entry values vary about as much as in the trained network (coefficient of variation versus ).

Second, the decoder. The trained decoder is close to a scaled pseudoinverse of the encoder (Frobenius cosine ). A network trained from scratch with its decoder tied to a scaled pseudoinverse of the encoder still reaches , and a freely-trained decoder on the three-scalar ansatz reaches only (Table 1). The larger cost of swapping in a pseudoinverse post hoc is therefore mostly because the trained encoder was tuned for its own decoder, not because a pseudoinverse decoder is inherently worse: when the encoder co-adapts, the constraint is nearly free.

Finally, it is unclear whether the trained network's binary support is special at all: in the three-scalar family it is not, as a designed biregular code slightly outperforms the three-scalar version of the trained code.

Together, these results account for most of the trained network's performance and pin down its main mechanism — a sparse binary code decoded by a near-pseudoinverse — leaving the encoder's specific per-entry values and the decoder's small residual structure as the pieces we have not fully explained.

Discussion and Future Work

Our reverse-engineered network looks like an empirical instance of the random sparse binary codes that Hänni et al. (2024) propose for computation in superposition: through training it independently arrives at the sparse-code-over-neurons motif their constructions predict. The match is suggestive rather than exact, as their analysis targets Boolean U-AND while our network computes continuous ReLUs.

The original motivation for this toy model was a testbed for parameter-decomposition methods such as APD (Braun et al., 2025) and SPD (Bushnaq et al., 2025). Now that we have elicited superposition and reverse-engineered the network, we can and will use it as a ground-truth solution against which to test these and other interpretability methods.

More broadly, we would like to know how far this picture extends: to deeper and more realistic networks, and to more realistic kinds of computation than the elementwise ReLU studied here. Does superposition still arise once there are several layers and bottlenecks, can we still reverse-engineer the resulting solution, and does anything resembling the error-correction layers of Hänni et al. (2024) appear?

Appendix A: The embedded model

The main text studies a model in which the ReLU neurons act directly on the input features. Braun et al. (2025)'s original compressed-computation model instead embeds the -dimensional input into through a fixed random matrix with unit-norm rows: the neurons read and write this -dimensional space through trainable and , and the output is read back out by the transpose . Because and has full row rank, the embedding is information-lossless: the network is equivalent to one acting directly on the input features through the effective weights and , which play exactly the roles of the encoder and decoder analyzed in the main text. It should therefore make no difference to the learned solution, and we confirm below that it does not. Figure 8 shows this embedded architecture, with the fixed embedding and unembedding restored relative to the axis-aligned model of Figure 1.

Figure 8. Embedded architecture (cf. Figure 1). The input is embedded as random near-orthogonal directions in a -dimensional space through a fixed matrix with unit-norm rows and read back out by the transpose ; the trainable encoder , ReLU neurons, and decoder are unchanged. The embedding (, ) is held fixed during training.

We retain the embedded variant because the compressed-computation toy model's original purpose was as a testbed for parameter-decomposition methods such as APD (Braun et al., 2025) and SPD (Bushnaq et al., 2025), for which the non-axis-aligned features of the embedded model are the more challenging and realistic setting. For the narrower claim that the trained network computes in superposition, the axis-aligned model in the main text is cleaner: with axis-aligned input features there is no random embedding that could, even in principle, be credited with the network's performance over the non-superposition baselines.

A.1 The embedded model reproduces the axis-aligned solution

We verify directly that adding the embedding changes nothing of substance. We train the embedded model under the same loss, optimizer, and schedule, and recompute every quantity from the main text. Table 2 reports the comparison. Across all of them — the per-feature loss spread, the codeword-length distribution, the swap test, the decoder's agreement with a pseudoinverse, and the three-scalar description — the two models are quantitatively the same; even the three fitted ansatz scalars agree to within . We conclude that the random embedding is incidental to the computation-in-superposition solution. Figures 9, 10, and 11 reproduce the main-text figures (Figures 2, 3, and 4) for the embedded model.

Table 2. The solution is unchanged by adding the random embedding. Headline quantities for the axis-aligned model (main text) and an otherwise identical model trained with Braun et al. (2025)'s random embedding. CV is the coefficient of variation of the per-feature MSE across the features. The ansatz ratio is the loss of the three-scalar description relative to the trained model.

Figure 9. Embedded model, per-feature MSE (cf. Figure 2). As in the axis-aligned model, the emulate-bias baseline and are bimodal (half the features handled, half left near the do-nothing baseline) while every exponent above spreads the error evenly across all features.

Figure 10. Embedded model, effective-encoder entries (cf. Figure 3). Pooled histogram of the entries of the effective encoder ; the same bimodal split into a small off-code cluster and a larger on-code cluster appears.

Figure 11. Embedded model, codeword regularity (cf. Figure 4). Large-entry counts per feature column and per neuron row for the effective encoder; the code is close to biregular, as in the axis-aligned model.

A.2 Transpose versus pseudoinverse unembedding

The embedded model reads its output back out with the transpose of the embedding. Because has unit-norm but not orthogonal rows, this transpose is only an approximate inverse, leaving a small residual cross-talk between the near-orthogonal feature directions. We retrain the embedded model with its unembedding set instead to the exact pseudoinverse — the left-inverse that removes this cross-talk — holding the loss, optimizer, schedule, embedding, and seed fixed. It reaches the same loss (within ) and the same per-feature error spread.

A.3 Other model sizes

To check that the solution is not specific to our , , choice, we vary the model size and measure the coefficient of variation of the per-feature MSE. Tightening the embedding to (keeping , ) still gives indications of superposition, though weaker: the coefficient of variation is , versus at . Scaling up an order of magnitude — features, a fixed random embedding into , and neurons — gives a low coefficient of variation again (). In both regimes the error is spread across all features rather than concentrated as in the naive solution. We check only this loss statistic here, not the full binary-code mechanism, so we report these as indications of superposition rather than confirmations.

Appendix B: Output attenuation

Following Bhagat et al. (2025), we examine the network's input–output response by driving a single feature with a value (all other features zero) and reading the output at index , repeating this for each of the features (Figure 12). An un-attenuated network would trace ; the trained network instead outputs a scaled-down ReLU, with mean slope on the active branch. The per-feature responses are nearly identical — the – band across features is too narrow to see — consistent with the even, superposition-like solution in which every feature is handled the same way.

This attenuation is a response to interference. Because features are sparse (active with probability ) and represented by overlapping codewords, an active feature's signal leaks through shared neurons into the outputs of the inactive features, pulling those outputs above the zero they should produce. Under this cross-talk is costly. The network compensates by attenuating every active output: shrinking the on-feature response reduces the leaked interference on the far more numerous inactive features, at the cost of a small under-shoot on the active one. The net effect is the systematic under-shoot in Figure 12.

Figure 12. Single-feature input–output response. For each feature in turn, feature alone is driven with value and we read the output at index ; the blue curve is the mean over all features. The trained network traces a ReLU attenuated to mean slope , below the true ReLU (dashed). The – band across features is too narrow to be visible: every feature is attenuated almost identically, as expected for the even superposition solution.

Appendix C: Codeword overlap minimization

When two features are active at once, their codewords interfere: neurons in both codewords receive contributions from both features, and the decoder must disentangle them. The size of this interference is set by the pairwise overlap , and under the largest overlaps dominate the loss. Before fitting the ansatz to a synthetic code we therefore reduce its overlaps, and prefer codes whose overlaps are both small and uniform, which minimizes the worst-case interference. We do this with a simple edge-swap procedure.

Let the pairwise overlap matrix be with . Our objective is the sum of off-diagonal squared overlaps, .

At each iteration: sample two edges uniformly at random from the edges in ; if , , , and , propose the swap removing and and adding and ; accept if the objective does not increase.

We run iterations per code. Edge swaps preserve codeword length (and, for biregular codes, neuron degree), so the structural properties of the family are kept fixed while the overlap distribution is optimized.