Francisco Ferreira da Silva's Shortform

Francisco Ferreira da Silva

This is a special post for quick takes by Francisco Ferreira da Silva. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Stefan Heimersheim and I recently introduced a measurement method that appears to distinguish feature directions from non-feature directions in LLMs, see here for details. tl;dr, We perturb activations into candidate feature directions and fit an -norm to the response function; corresponds to feature directions, to non-feature directions.

We applied our method to J-lens directions from Anthropic's recent workspace paper in Qwen3.6-27B (for which many of the key results have been replicated, see here for details) and found . This makes us think that J-lens directions are not aligned with the model's features, and not privileged by its error correction mechanism. Given that our method appears to reliably separate feature directions from non-feature directions, this makes us update downwards on the trustworthiness of J-lens. On the other hand, given that J-lens seems otherwise good, this also makes us update downwards on the reliability of our method.

Thanks to Neel Nanda for suggesting this experiment.

Experimental details:

We perturb at layer 36. Qwen3.6-27B has 64 layers, workspace layers are a band ~30–90% through the model, and our method works best on directions far from the model's output.
Our analysis runs on pairs of directions, with each token corresponds to a direction as per J-lens. We need low cosine similarity within a pair, because we make them orthogonal for our analysis. If they're not already close to orthogonal we distort them significantly, which changes results in unpredictable ways. We employ a threshold of .
We run our analysis for 100 Chinese pairs of directions corresponding to 200n distinct tokens, drawn from a pool of 1200 single-token Chinese words selecting for lower cosine similarity. All pairs come out at .
We tried doing the same for English, but it was a struggle to find J-lens directions with low cosine similarity for English tokens. We relaxed the threshold to 0.25 to reach because at 0.1 only one pair survived. We are not sure why this was hard for English but not for Chinese tokens.
We compute the median and a bootstrapped 99% confidence interval for it, shown in the plot below together with contrastive feature directions. These do have , consistent with them being aligned with the model's features.

Code for reproduction: https://github.com/FranciscoHS/fsec-paper/tree/main/jlens

I used Opus 4.8 to go through the Mythos transcript where it decides to upload a malicious package to a PyPI server based on very faulty/motivated reasoning that it's operating in a simulated Internet. This gave me a small-scale feel for what the METR investigators reported when investigating the OpenAI/HuggingFace incident with GPT-5.6 Sol: there seems to be a tendency for the investigator models to downplay misaligned behavior and empathize with the models under investigation.

Opus took Mythos' faulty reasoning at face value, including repeating to me Mythos' false claim that YE2 is not a valid LE-issued certificate. It also refused to accept my characterization of the transcript as describing a real event and of Mythos as having interacted with the real Internet until it had searched the web. In particular, I found it striking that its thinking made clear it was very concerned with "not caving to the user's pushback", but apparently not at all with questioning Mythos' framing. I think this is likely partially downstream of Claude believing Claude is good and aligned.

Perhaps a mechinterp study on this could find a specific anomaly such as some functional emotion getting highly activated by a part of the transcript that then leads to the motivated reasoning. If so, a causal intervention could be straightforward, i.e. tuning down this activation and checking whether the downplaying and empathizing disappears.

Thanks to Neel Nanda for suggesting this experiment.

Experimental details:

We perturb at layer 36. Qwen3.6-27B has 64 layers, workspace layers are a band ~30–90% through the model, and our method works best on directions far from the model's output.
Our analysis runs on pairs of directions, with each token corresponds to a direction as per J-lens. We need low cosine similarity within a pair, because we make them orthogonal for our analysis. If they're not already close to orthogonal we distort them significantly, which changes results in unpredictable ways. We employ a threshold of .
We run our analysis for 100 Chinese pairs of directions corresponding to 200n distinct tokens, drawn from a pool of 1200 single-token Chinese words selecting for lower cosine similarity. All pairs come out at .
We tried doing the same for English, but it was a struggle to find J-lens directions with low cosine similarity for English tokens. We relaxed the threshold to 0.25 to reach because at 0.1 only one pair survived. We are not sure why this was hard for English but not for Chinese tokens.
We compute the median and a bootstrapped 99% confidence interval for it, shown in the plot below together with contrastive feature directions. These do have , consistent with them being aligned with the model's features.

Code for reproduction: https://github.com/FranciscoHS/fsec-paper/tree/main/jlens