Francisco Ferreira da Silva's Shortform

Francisco Ferreira da Silva

This is a special post for quick takes by Francisco Ferreira da Silva. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Stefan Heimersheim and I recently introduced a measurement method that appears to distinguish feature directions from non-feature directions in LLMs, see here for details. tl;dr, We perturb activations into candidate feature directions and fit an -norm to the response function; corresponds to feature directions, to non-feature directions.

We applied our method to J-lens directions from Anthropic's recent workspace paper in Qwen3.6-27B (for which many of the key results have been replicated, see here for details) and found . This makes us think that J-lens directions are not aligned with the model's features, and not privileged by its error correction mechanism. Given that our method appears to reliably separate feature directions from non-feature directions, this makes us update downwards on the trustworthiness of J-lens. On the other hand, given that J-lens seems otherwise good, this also makes us update downwards on the reliability of our method.

Thanks to Neel Nanda for suggesting this experiment.

Experimental details:

We perturb at layer 36. Qwen3.6-27B has 64 layers, workspace layers are a band ~30–90% through the model, and our method works best on directions far from the model's output.
Our analysis runs on pairs of directions, with each token corresponds to a direction as per J-lens. We need low cosine similarity within a pair, because we make them orthogonal for our analysis. If they're not already close to orthogonal we distort them significantly, which changes results in unpredictable ways. We employ a threshold of .
We run our analysis for 100 Chinese pairs of directions corresponding to 200n distinct tokens, drawn from a pool of 1200 single-token Chinese words selecting for lower cosine similarity. All pairs come out at .
We tried doing the same or English, but it was a struggle to find J-lens directions with low cosine similarity for English tokens. We relaxed the threshold to 0.25 to reach because at 0.1 only one pair survived. We are not sure why this was hard for English but not for Chinese tokens.
We compute the median and a bootstrapped 99% confidence interval for it, shown in the plot below together with contrastive feature directions. These do have , consistent with them being aligned with the model's features.

Code for reproduction: https://github.com/FranciscoHS/fsec-paper/tree/main/jlens

Thanks to Neel Nanda for suggesting this experiment.

Experimental details:

We perturb at layer 36. Qwen3.6-27B has 64 layers, workspace layers are a band ~30–90% through the model, and our method works best on directions far from the model's output.
Our analysis runs on pairs of directions, with each token corresponds to a direction as per J-lens. We need low cosine similarity within a pair, because we make them orthogonal for our analysis. If they're not already close to orthogonal we distort them significantly, which changes results in unpredictable ways. We employ a threshold of .
We run our analysis for 100 Chinese pairs of directions corresponding to 200n distinct tokens, drawn from a pool of 1200 single-token Chinese words selecting for lower cosine similarity. All pairs come out at .
We tried doing the same or English, but it was a struggle to find J-lens directions with low cosine similarity for English tokens. We relaxed the threshold to 0.25 to reach because at 0.1 only one pair survived. We are not sure why this was hard for English but not for Chinese tokens.
We compute the median and a bootstrapped 99% confidence interval for it, shown in the plot below together with contrastive feature directions. These do have , consistent with them being aligned with the model's features.

Code for reproduction: https://github.com/FranciscoHS/fsec-paper/tree/main/jlens