This is a replication of Adversarial Attacks Leverage Interference Between Features in Superposition, completed as part of the Second Look Summer Fellowship.

tl;dr

• In a previous blog we replicated the toy models examples from Stevenson et al. In this work, we reproduce the vision transformer experiments from the paper:
  • On every dataset, we find that increases in superposition lead to adversarial attacks that line up more closely with class-vector geometry. Likewise, trends across a variety of metrics we measure match the paper.
• Beyond the paper, we find that:
  • The coupling between the input and classification head geometry is independent of adversarial attack methods.
  • Adversarial training increased attack robustness but we couldn’t measure changes in geometry using the methods from the paper.
  • Likewise, we retrained models of different widths but we didn’t measure a change in coupling with geometry and robustness.

Our code is here, models are here, and we encourage readers to fork/run followup experiments!

Background

Elhage et al., 2022 defines superposition as the phenomenon where “models represent more features than they have dimensions.” Stevinson et al., argues this interference is a mechanism behind adversarial examples.

In our previous work, we replicated experiments on Stevinson et al. on toy models. In this work, we investigate three key claims on Stevinson et al. on ViTs: 1. attacks can be predicted by latent interference, 2. the coupling strengthens as the bottleneck tightens, 3. Interference patterns are consistent across seeds.

Models and Datasets

While in the previous post we studied toy-autoencoders, in this post we study the largest open-source image classification network: vision transformer (ViT). To control superposition in the network, Stevinson et al. adds an engineered bottleneck layer before the classification head. As shown in Figure 1, a 12-layer ViT produces a pooled classification token ∈ . Then, Stevinson et al. adds a linear encoder ✕ compresses it to m dimensions, before using a linear decoder ✕ to read k class logits back out. 

Each decoder row corresponds to a unique class so we can take tells us how much any two class directions overlap (we call this interference). When m < k, reads out more logits than there are dimensions in , so rows of are forced to overlap causing interference.

The experiments use four image classification datasets: CIFAR-10/100, STL-10, and Tiny-ImageNet.

Metrics 

Adversarial attacks are computed by projected gradient descent (PGD) (Madry et al., 2018). For each successful attack, there are two measurements that serve as building blocks for the metrics we report: input-space profile and latent profile.

Input-space profile

gives the gradient of the logit for class c with respect to the original image x. In other words, tells us what direction each pixel value in the image would have to move to increase the logit for class c. is the difference between the adversarial example and the original image: that is, The input-space profile for class

measures how the perturbation pushes each class logit. Because gives the direction in the original image that would most increase the logit for class , a_c  can be read as a heuristic for how much a attacks a specific class.

Latent profile

The latent profile for class , , measures how the class feature vector aligns with the attack direction . 

Metrics

We report metrics used by Stevenson et al. for clean comparison with our results:

1. Correlation metric : mean Pearson correlation between and across 5 seeds. The authors exclude source and target classes because we expect and and therefore and are trivially correlated. If the input-space and latent profiles should agree with a high Pearson correlation, this is some evidence that attacks exploit interference.  
2. Interference magnitude : where is the interference matrix across classes. This gives the mean interference over all unique pairs of non-identical rows.
3. Cross-seed consistency : the Pearson correlation between interference matrices across seeds.

Interference Predicts Attacks as Bottleneck Tightens

We replicate four bottleneck sweeps on each dataset and compare our results with paper. Attack samples are generated using for 50 steps and 5 restarts. All metrics are reported with mean ± std over 5 seeds. 

We found that as decreases (bottleneck tightens), the clean accuracy of the models drops and rises, which signals less capacity and more interference. decreases monotonically with , which supports the hypothesis that latent interference predicts adversarial attacks.

Superposition Without Artificial Bottleneck

Stevinson et al. also removes the dimensional bottleneck from the ViTs and control superposition through weight decay. Weight decay pushes weights toward zero, and this drives the learned representation toward low rank. In this way, the model has less effective dimensions which creates superposition. The effective dimensions is estimated by the number of principle components that explain 95% of the variance.

As shown in Table 1, as weight decay gradually increases, the model has less effective dimensions and clean accurary monotonically drops as expected. The fewer dimensions that the models have, the stronger the coupling between the input-space and output profile becomes. The hypothesis holds in ordinary models without the artificial bottleneck (i.e., models where m=k). 

Table 1: Weight Decay affects model capacity on CIFAR-10

Coupling Is Attack-independent

The mainline is measured on PGD perturbations (gradient based) and the input profile which is also gradient-based. Because they are constructed by gradients from the same model, a skeptic may ask whether the correlation is partly self-fulfilling. 

Therefore, we explored two more attack families on identical models and the CIFAR-10 dataset: (1) Square, a black-box random search attack with no gradient calculation, (2) CW, an gradient-based attack that, instead of spending a fixed budget, searches for the smallest perturbation that flips the prediction.

All three families have higher correlation for a stricter bottleneck which collapses as . The fact that gradient-free Square attack follows the same trend shows that the coupling between attacks and latent interference is not an artifact of the attack algorithm.

Geometry metrics don’t change with adversarial training

As an extension, we were curious about the mechanism behind adversarial training (AT) that makes the model more robust. Is it because AT optimizes the latent geometry so that features are arranged more orthogonal to one another? Does it break the correlation between input space and latent space attack profile?

We run Madry PGD-AT on the same models and CIFAR-10 dataset. As shown in Table 2, robust accuracy jumps ~5× at every ratio, yet interference and effective dimension m̄ are unchanged. Separately, we observe that does not decrease with AT, implying attacks that remain geometry-aligned. Therefore, we believe that AT hardens the backbone features that reach the attack surface rather than untangling the head of the models. 

Table 2: Results on adversarially trained models

Gorton et Lewis, 2025, also test if adversarial training reduces superposition. They test the SAE reconstruction loss as a proxy for superposition, broadly motivated by the idea that SAEs aren’t able to model a large number of co-activating features (for more details see section 4.2 in Gorton et Lewis, 2025). Contrasting our findings, the authors find that more robust models achieve lower SAE reconstruction losses which is evidence that the training is reducing the amount of superposition in the model. We don’t find evidence of superposition decreasing across our mentics, but the methodology is different and is only calculated over the bottleneck at the end of the model.

Capacity Changes Input and Latent Space Correlation

If superposition causes vulnerability, giving the network more room should relieve both. So we remove the bottleneck and increase the width of ViT at fixed depth 12 — , heads = , MLP = , 5 seeds — so ambient capacity per class grows from to .

Table 3: The effect of ViT width on robustness and head interference

As shown in Table 3, capacity changes latent and input coupling but not superposition nor robustness. Interference is flat at across all widths, which means that the head packs its class vectors just as tightly however much ambient room it gets (m̄ ≥ 19 ≫ k = 10 everywhere, so no width in the sweep puts dimensional pressure on the class vectors). Notably, falls from 0.22 to 0.09 showing that input-space attacks decouple from the head geometry as the model gains capacity. This could suggest that a wider network gives a perturbation more input-space routes that do not mirror the head’s interference pattern. 

Discussion

Our extension on the paper suggests that AT affects robustness but final layer geometry and coupling are left unchanged. Likewise, model width moved the  final layer coupling with robustness and bottleneck geometry unchanged. These experiments show it is possible for robustness and head geometry or coupling strength to move independently, which is perhaps to be expected. Studying superposition outside of toy models is inherently difficult because we cannot interfere directly nor directly measure superposition, making causal claims difficult and measurements inherently limited. Our current measurements are useful tools which can provide compelling evidence but aren’t able to be a reliable source of ground truth. We believe future work in developing new metrics and interventions is especially promising.