What’s up with LLMs representing XORs of arbitrary features?

[-]Hoagy2yΩ20362

My hypothesis about what's going on here, apologies if it's already ruled out, is that we should not think of it separately computing the XOR of A and B, but rather that features A and B are computed slightly differently when the other feature is off or on. In a high dimensional space, if the vector and the vector $(A | \neg B)$ are slightly different, then as long as this difference is systematic, this should be sufficient to successfully probe for $A \oplus B$ .

For example, if A and B each rely on a sizeable number of different attention heads to pull the information over, they will have some attention heads which participate in both of them, and they would 'compete' in the softmax, where if head C is used in both writing features A and B, it will contribute less to writing feature A if it is also being used to pull across feature B, and so the representation of A will be systematically different depending on the presence of B.

It's harder to draw the exact picture for MLPs but I think similar interdependencies can occur there though I don't have an exact picture of how, interested to discuss and can try and sketch it out if people are curious. Probably would be like, neurons will participate in both, neurons which participate in A and B will be more saturated if B is active than if B is not active, so the output representation of A will be somewhat dependent on B.

More generally, I expect the computation of features to be 'good enough' but still messy and somewhat dependent on which other features are present because this kludginess allows them to pack more computation into the same number of layers than if the features were computed totally independently.

[-]Sam Marks2yΩ120

Neat hypothesis! Do you have any ideas for how one would experimentally test this?

[-]Hoagy2yΩ110

Well the substance of the claim is that when a model is calculating lots of things in superposition, these kinds of XORs arise naturally as a result of interference, so one thing to do might be to look at a small algorithmic dataset of some kind where there's a distinct set of features to learn and no reason to learn the XORs and see if you can still probe for them. It'd be interesting to see if there are some conditions under which this is/isn't true, e.g. if needing to learn more features makes the dependence between their calculation higher and the XORs more visible.

Maybe you could also go a bit more mathematical and hand-construct a set of weights which calculates a set of features in superposition so you can totally rule out any model effort being expended on calculating XORs and then see if they're still probe-able.

Another thing you could do is to zero-out or max-ent the neurons/attention heads that are important for calculating the feature, and see if you can still detect an $A \oplus B$ feature. I'm less confident in this because it might be too strong and delete even a 'legitimate' $A \oplus B$ feature or too weak and leave some signal in.

This kind of interference also predicts that the $A | B$ and $A | \neg B$ features should be similar and so the degree of separation/distance from the category boundary should be small. I think you've already shown this to some extent with the PCA stuff though some quantification of the distance to boundary would be interesting. Even if the model was allocating resource to computing these XORs you'd still probably expect them to be much less salient though so not sure if this gives much evidence either way.

[-]Clément Dumas2y10

Would you expect that we can extract xors from small models like pythia-70m under your hypothesis?

[-]Hoagy2y10

Yeah I'd expect some degree of interference leading to >50% success on XORs even in small models.

[-]Clément Dumas2y10

You can get ~75% just by computing the or. But we found that only at the last layer and step16000 of Pythia-70m training it achieves better than 75%, see this video

[-]Fabien Roger2y*Ω20351

On xor being represented incidentally:

I find experiments where you get <<50% val acc sketchy so I quickly ran my own using a very fake dataset made out of vectors in {-1,1}^d that I pass through 10 randomly initialized ReLU MLPs with skip connections. Here, the "features" I care about are canonical directions in the input space.

What I find:

XOR is not incidentally represented if the features are not redundant
XOR is incidentally represented if the features are very redundant (which is often the case in Transformers, but maybe not to the extent needed for XOR to be incidentally represented). I get redundancy by using input vectors that are a concatenation of many copies of a smaller input vector.

See my code for more details: https://pastebin.com/LLjvaQLC

[-]Adam Shai2y32

I think you might need to change permissions on your github repository?

[-]Fabien Roger2y20

Oops, here is the fixed link:

https://pastebin.com/LLjvaQLC

[+][comment deleted]2y10

[-]gwern2y*Ω11249

This is wacky but seems like a plausible thing a model might do: by doing this, the model would be able to, in later layers, make use of arbitrary boolean functions of early layer features.

This seems less wacky from a reservoir computing perspective and from the lack of recurrency: you have no idea early on what you need*, so you compute a lot of random complicated stuff up front, which is then available for the self-attention in later layers to do gradient descent on to pick out the useful features tailored to that exact prompt, and then try to execute the inferred algorithm in the time left. 'arbitrary boolean functions of early layer features' is not necessarily something I'd have expected the model to compute, but when put that way, it does seem like a bunch of those hanging around could be useful, no?

This is analogous to all the interpretability works which keep finding a theme of 'compute or retrieve everything which might be relevant up front, and then throw most of it away to do some computation with the remainder and make the final prediction'. Which is further consistent with the observations about sparsity & pruning: if most of a NN is purely speculative and exists to enable learning when one of those random functions turns out to be useful, then once learning has been accomplished, of course you can stop screwing around with speculatively computing loads of useless random things.

EDIT: if this perspective is 'the reason', I guess it would predict that at very low loss or perfect prediction, these xor features would go away unless they had a clearcut use. And that if you trained in a sparsity-creating fashion or did knowledge-distillation, then the model would also be incentivized to get rid of them. You would also maybe be able to ablate/delete them, in some way, and show that transfer learning and in-context learning become worse? I would expect any kind of recurrent or compressive approach would tend to do less of every kind of speculative computation, so maybe that would work too.

* anything in the early layers of a Transformer has no idea what is going on, because you have no recurrent state telling you anything from previous timesteps which could summarize the task or relevant state for you; and the tokens/activations are all still very local because you're still early in the forward pass. This is part of why I think the blind spot might be from early layers throwing away data prematurely, making irreversible errors.

[-]eggsyntax8mo20

and the tokens/activations are all still very local because you're still early in the forward pass

I don't understand why this would necessarily be true, since attention heads have access to values for all previous token positions. Certainly, there's been less computation at each token position in early layers, so I could imagine there being less value to retrieving information from earlier tokens. But on the other hand, I could imagine it sometimes being quite valuable in early layers just to know what tokens had come before.

[-]leogao2yΩ5170

I'm having some trouble replicating this result in a not exactly comparable setting (internal model, looking at is_alice xor amazon_sentiment). I get 90%+ on the constituent datasets, but only up to 75% on the xor depending on which layer I look at.

(low confidence, will update as I get more results)

[-]Sam Marks2yΩ120

Thanks for doing this! Can you share the dataset that you're working with? I'm traveling right now, but when I get a chance I might try to replicate your failed replication on LLaMA-2-13B and with my codebase (which can be found here; see especially xor_probing.ipynb).

[-]leogao2yΩ110

The training set is a random 100k subsample of this dataset: https://huggingface.co/datasets/amazon_polarity

I'm prepending Alice/Bob and doing the xor of the label in exactly the same way you do.

[-]ryan_greenblatt2yΩ8165

I reran my experiments from above on a “reset” version of LLaMA-2-13B. What this means is that, for each parameter in LLaMA-2-13B, I shuffled the weights of that parameter by permuting them along the last dimension

Why do you get <50% accuracy for any of the categories? Shouldn't a probe trained on any reasonable loss function always get >50% accuracy on any binary classification task?

[-]Sam Marks2yΩ23-5

I'm not really sure, but I don't think this is that surprising. I think when we try to fit a probe to "label" (the truth value of the statement), this is probably like fitting a linear probe to random data. It might overfit on some token-level heuristic which is ideosyncratically good on the train set but generalizes poorly to the val set. E.g. if disproportionately many statements containing "India" are true on the train set, then it might learn to label statements containing "India" as true; but since in the full dataset, there is no correlation between "India" and being true, correlation between "India" and true in the val set will necessarily have the opposite sign.

[-]ryan_greenblatt2yΩ120

Are the training and val sets not IID? Are they small enough that we either get serious overfit or huge error bars?

[-]ryan_greenblatt2yΩ120

If the datasets are IID and large and the loss function is reasonable, then if there is just noise, the probe should learn to just always predict the more common class and not have any variance. This should always result in >50% accuracy.

[-]Sam Marks2yΩ790

There's 1500 statements in each of cities and neg_cities, and LLaMA-2-13B has residual stream dimension 5120. The linear probes are trained with vanilla logistic regression on {80% of the data in cities} \cup {80% of the data in neg_cities} and the accuracies reported are evaluated on {remaining 20% of the data in cities} \cup {remaining 20% of the data in neg_cities}.

So, yeah, I guess that the train and val sets are drawn from the same distribution but are not independent (because of the issue I mentioned in my comment above). Oops! I guess I never thought about how with small datasets, doing an 80/20 train/test split can actually introduce dependencies between the train and test data. (Also yikes, I see people do this all the time.)

Anyway, it seems to me that this is enough to explain the <50% accuracies -- do you agree?

[-]Sam Marks2yΩ8142

Using a dataset of 10,000 inputs of the form
[random LLaMA-13B generated text at temperature 0.8] [either the most likely next token or the 100th most likely next token, according to LLaMA-13B] ["true" or "false"] ["banana" or "shed"]
I've rerun the probing experiments. The possible labels are

has_true: is the second last token "true" or "false"?
has_banana: is the last token "banana" or "shed"?
label: is the third last token the most likely or the 100th most likely?

(this weird last option is because I'm adapting a dataset from the Geometry of Truth paper about likely vs. unlikely text).

Here are the results for LLaMA-2-13B

And here are the results for the reset network

I was a bit surprised that the model did so badly on has_true, but in hindsight, considering that the activations are extracted over the last token and "true"/"false" is the penultimate token, this seems fine.

Mostly I view this as a sanity check to make sure that when the dataset is larger we don't get the <<50% probe accuracies. I think to really dig into this more, one would need to do this with features which are not token-level and which are unambiguously linearly accessible (unlike the "label" feature here).

@ryan_greenblatt @abhatt349 @Fabien Roger

[-]ryan_greenblatt2yΩ230

Yep I think I agree, I didn't understand the point you made about systematic anti-correlation originally.

If I understand correctly the issues is something like:

There are 10 India related statements, exactly 5 of which are false and 5 of which are true.
We do a random split of all the data, so if there is more true+india in train there will be more false+india in test

There are of course various fixes to make the data actually IID.

[-]Aryan Bhatt2y3-2

If it's easy enough to run, it seems worth re-training the probes exactly the same way, except sampling both your train and test sets with replacement from the full dataset. This should avoid that issue. It has the downside of allowing some train/test leakage, but that seems pretty fine, especially if you only sample like 500 examples for train and 100 for test (from each of cities and neg_cities).

I'd strongly hope that after doing this, none of your probes would be significantly below 50%.

[-]Rohin Shah2y*Ω9147

Nice post, and glad this got settled experimentally! I think it isn't quite as counterintuitive as you make it out to be -- the observations seem like they have reasonable explanations.

I feel pretty confident that there's a systematic difference between basic features and derived features, where the basic features are more "salient" -- I'll be assuming such a distinction in the rest of the comment.

(I'm saying "derived" rather than "XOR" because it seems plausible that some XOR features are better thought of as "basic", e.g. if they were very useful for the model to compute. E.g. the original intuition for CCS is that "truth" is a basic feature, even though it is fundamentally an XOR in the contrast pair approach.)

For the more mechanistic explanations, I want to cluster them into two classes of hypotheses:

Incidental explanations: Somehow "high-dimensional geometry" and "training dynamics" means that by default XORs of basic features end up being linearly represented as a side effect / "by accident". I think Fabien's experiments and Hoagy's hypothesis fit here.
1. I think most mechanistic explanations here will end up implying a decay postulate that says "the extent to which an incidental-XOR happens decays as you have XORs amongst more and more basic features". This explains why basic features are more salient than derived features.
Utility explanations: Actually it's often quite useful for downstream computations to be able to do logical computations on boolean variables, so during training there's a significant incentive to represent the XOR to make that happen.
1. Here the reason basic features are more salient is that basic features are more useful for getting low loss, and so the model allocates more of its "resources" to those features. For example, it might use more parameter norm (penalized by weight decay) to create higher-magnitude activations for the basic features.

I think both of the issues you raise have explanations under both classes of hypotheses.

Exponentially many features:

An easy counting argument shows that the number of multi-way XORs of N features is ~. [...] There are two ways to resist this argument, which I’ll discuss in more depth later in “What’s going on?”:
To deny that XORs of basic features are actually using excess model capacity, because they’re being represented linearly “by accident” or as an unintended consequence of some other useful computation. (By analogy, the model automatically linearly represents ANDs of arbitrary features without having to expend extra capacity.)
To deny forms of RAX that imply multi-way XORs are linearly represented, with the model somehow knowing to compute $a \oplus b$ and $a \oplus c$ , but not $a \oplus b \oplus c$ .

While I think the first option is possible, my guess is that it's more like the second option.

On incidental explanations, this is explained by the decay postulate. For example, maybe once you hit 3-way XORs, the incidental thing is much less likely to happen, and so you get ~ $\frac{N^{2}}{2}$ pairwise XORs instead of the full ~ $2^{N}$ set of multi-way XORs.

On utility explanations, you would expect that multi-way XORs are much less useful for getting low loss than two-way XORs, and so computation for multi-way XORs is never developed.

Generalization:

logistic regression on the train set would learn the direction $v_{a} + v_{a \oplus b}$ where $v_{f}$ is the direction representing a feature f. [...] the argument above would predict that linear probes will completely fail to generalize from train to test. This is not the result that we typically see [...]
One of these assumptions involves asserting that “basic” feature directions (those corresponding to a and b) are “more salient” than directions representing XORs – that is, the variance along $v_{a}$ and $v_{b}$ is larger than variance along $v_{a \oplus b}$ . However, I’ll note that:
it’s not obvious why something like this would be true, suggesting that we’re missing a big part of the story for why linear probes ever generalize;
even if “basic” feature directions are more salient, the argument here still goes through to a degree, implying a qualitatively new reason to expect poor generalization from linear probes.

For the first point I'd note that (1) the decay postulate for incidental explanations seems so natural and (2) the "derived features are less useful than basic features and so have less resources allocated to them" seems sufficient for utility explanations.

For the second point, I'm not sure that the argument does go through. In particular you now have two possible outs:

Maybe if $v_{a}$ is twice as salient as $v_{a \oplus b}$ , you learn a linear probe that is entirely $v_{a}$ , or close enough to it (e.g. if it is exponentially closer). I'd guess this isn't the explanation, but I don't actually know what linear probe learning theory predicts here.
Even if you do learn $v_{a} + v_{a \oplus b}$ , it doesn't seem obvious that test accuracy should be < 100%. In particular, if $v_{a}$ is more salient by having activations that are twice as large, then it could be that even when b flips from 0 to 1 and $v_{a \oplus b}$ is reversed, $v_{a}$ still overwhelms $v_{a \oplus b}$ and so every input is still classified correctly (with slightly less confidence than before).

On the other hand, RAX introduces a qualitatively new way that linear probes can fail to learn good directions. Suppose a is a feature you care about (e.g. “true vs. false statements”) and b is some unrelated feature which is constant in your training data (e.g. b = “relates to geography”). [...]
This is wild. It implies that you can’t find a good direction for your feature unless your training data is diverse with respect to every feature that your LLM linearly represents.

Fwiw, failures like this seem plausible without RAX as well. We explicitly make this argument in our goal misgeneralization paper (bottom of page 9 / Section 4.2), and many of our examples follow this pattern (e.g. in Monster Gridworld, you see a distribution shift from "there is almost always a monster present" in training to "there are no monsters present" at test time).

I agree strong RAX without any saliency differences between features would imply this problem is way more widespread than it seems to be in practice, but I don't think it's a qualitatively new kind of generalization failure (and also I think strong RAX without saliency differences is clearly false).

Maybe models track which features are basic and enforce that these features be more salient
In other words, maybe the LLM is recording somewhere the information that a and b are basic features; then when it goes to compute $a \oplus b$ , it artificially makes this direction less salient. And when the model computes a new basic feature as a boolean function of other features, it somehow notes that this new feature should be treated as basic and artificially increases the salience along the new feature direction.

I don't think the model has to do any active tracking; on both hypotheses this happens by default (in incidental explanations, because of the decay postulate, and in utility explanations, because the $a \oplus b$ feature is less useful and so fewer resources go towards computing it).

[-]Sam Marks2yΩ340

I agree with a lot of this, but some notes:

Exponentially many features
[...]
On utility explanations, you would expect that multi-way XORs are much less useful for getting low loss than two-way XORs, and so computation for multi-way XORs is never developed.

The thing that's confusing here is that the two-way XORs that my experiments are looking at just seem clearly not useful for anything. So I think any utility explanation that's going to be correct needs to be a somewhat subtle one of the form "the model doesn't initially know which XORs will be useful, so it just dumbly computes way more XORs than it needs, including XORs which are never used in any example in training." Or in other words "the model has learned the algorithm 'compute lots of XORs' rather than having learned specific XORs which it's useful to compute."

I think this subtlety changes the story a bit. One way that it changes the story is that you can't just say "the model won't compute multi-way XORs because they're not useful" -- the two-way XORs were already not useful! You instead need to argue that the model is implementing an algorithm which computed all the two-way XORs but didn't compute XORs of XORs; it seems like this algorithm might need to encode somewhere information about which directions correspond to basic features and which don't.

On the other hand, RAX introduces a qualitatively new way that linear probes can fail to learn good directions. Suppose a is a feature you care about (e.g. “true vs. false statements”) and b is some unrelated feature which is constant in your training data (e.g. b = “relates to geography”). [...]
Fwiw, failures like this seem plausible without RAX as well. We explicitly make this argument in our goal misgeneralization paper (bottom of page 9 / Section 4.2), and many of our examples follow this pattern (e.g. in Monster Gridworld, you see a distribution shift from "there is almost always a monster present" in training to "there are no monsters present" at test time).

Even though on a surface level this resembles the failure discussed in the post (because one feature is held fixed during training), I strongly expect that the sorts of failures you cite here are really generalization failure for "the usual reasons" of spurious correlations during training. For example, during training (because monsters are present), "get a high score" and "pick up shields" are correlated, so the agents learn to value picking up shields. I predict that if you modified the train set so that it's no longer useful to pick up shields (but monsters are still present), then the agent would no longer pick up shields, and so would no longer misgeneralize in this particular way.

In contrast, the point I'm trying to make in the post is that RAX can cause problems even in the absence of spurious correlations like this.^[1]

I don't think the model has to do any active tracking; on both hypotheses this happens by default (in incidental explanations, because of the decay postulate, and in utility explanations, because the feature is less useful and so fewer resources go towards computing it).

As you noted, it will sometimes be the case that XOR features are more like basic features than derived features, and thus will be represented with high salience. I think incidental hypotheses will have a really hard time explaining this -- do you agree?

For utility hypotheses, the point is that there needs to be something different in model internals which says "when computing these features represent the result with low salience, but when computing these features represent the result with high salience." Maybe on your model this is something simple like the weights computing the basic features being larger than weights computing derived features? If so, that's the tracking I'm talking about, and is a potential thread to pull on for distinguishing basic vs. derived features using model internals.

^{^}
If you want you could rephrase this issue as " $a$ and $a \oplus b$ are spuriously correlated in training," so I guess I should say "even in the absence of spurious correlations among basic features."

[-]Rohin Shah2yΩ340

The thing that's confusing here is that the two-way XORs that my experiments are looking at just seem clearly not useful for anything.

Idk, I think it's pretty hard to know what things are and aren't useful for predicting the next token. For example, some of your features involve XORing with a "has_not" feature -- XORing with an indicator for "not" might be exactly what you want to do to capture the effect of the "not".

(Tbc here the hypothesis could be "the model computes XORs with has_not all the time, and then uses only some of them", so it does have some aspect of "compute lots of XORs", but it is still a hypothesis that clearly by default doesn't produce multiway XORs.)

In contrast, the point I'm trying to make in the post is that RAX can cause problems even in the absence of spurious correlations like this.^[1]
^{^}
If you want you could rephrase this issue as " and $a \oplus b$ are spuriously correlated in training," so I guess I should say "even in the absence of spurious correlations among basic features."

... That's exactly how I would rephrase the issue and I'm not clear on why you're making a sharp distinction here.

As you noted, it will sometimes be the case that XOR features are more like basic features than derived features, and thus will be represented with high salience. I think incidental hypotheses will have a really hard time explaining this -- do you agree?

I mean, I'd say the ones that are more like basic features are like that because it was useful, and it's all the other XORs that are explained by incidental hypotheses. The incidental hypotheses shouldn't be taken to be saying that all XORs are incidental, just the ones which aren't explained by utility. Perhaps a different way of putting it is that I expect both utility and incidental hypotheses to be true to some extent.

Maybe on your model this is something simple like the weights computing the basic features being larger than weights computing derived features? If so, that's the tracking I'm talking about, and is a potential thread to pull on for distinguishing basic vs. derived features using model internals.

Yes, on my model it could be something like the weights for basic features being large. It's not necessarily that simple, e.g. it could also be that the derived features are in superposition with a larger number of other features that leads to more interference. If you're calling that "tracking", fair enough I guess; my main claim is that it shouldn't be surprising. I agree it's a potential thread for distinguishing such features.

[-]Sam Marks2yΩ120

Idk, I think it's pretty hard to know what things are and aren't useful for predicting the next token. For example, some of your features involve XORing with a "has_not" feature -- XORing with an indicator for "not" might be exactly what you want to do to capture the effect of the "not".

I agree that "the model has learned the algorithm 'always compute XORs with has_not'" is a pretty sensible hypothesis. (And might be useful to know, if true!) FWIW, the stronger example of "clearly not useful XORs" I was thinking of has_true XOR has_banana, where I'm guessing you're anticipating that this XOR exists incidentally.

If you want you could rephrase this issue as " and $a \oplus b$ are spuriously correlated in training," so I guess I should say "even in the absence of spurious correlations among basic features."
... That's exactly how I would rephrase the issue and I'm not clear on why you're making a sharp distinction here.

Focusing again on the Monster gridworld setting, here are two different ways that your goals could misgeneralize:

player_has_shield is spuriously correlated with high_score during training, so the agent comes to value both
monster_present XOR high_score is spuriously correlated with high_score during training, so the agent comes to value both.

These are pretty different things that could go wrong. Before realizing that these crazy XOR features existed, I would only have worried about (1); now that I know these crazy XOR features exist ... I think I mostly don't need to worry about (2), but I'm not certain and it might come down to details about the setting. (Indeed, your CCS challenges work has shown that sometimes these crazy XOR features really can get in the way!)

I agree that you can think of this issue as just being the consequence of the two issues "there are lots of crazy XOR features" and "linear probes can pick up on spurious correlations," I guess this issue feels qualitatively new to me because it just seems pretty untractable to deal with it on the data augmentation level (how do you control for spurious correlations with arbitrary boolean functions of undesired features?). I think you mostly need to hope that it doesn't matter (because the crazy XOR directions aren't too salient) or come up with some new idea.

I'll note that if it ends up these XOR directions don't matter for generalization in practice, then I start to feel better about CCS (along with other linear probing techniques).^[1]

my main claim is that it shouldn't be surprising

If I had to articulate my reason for being surprised here, it'd be something like:

I didn't expect LLMs to compute many XORs incidentally
I didn't expect LLMs to compute many XORs because they are useful

but lots of XORs seem to get computed anyway. So at least one of these two mechanisms is occurring a surprising (to me) amount. If there's a lot more incidental computation, then why? (Based on Fabian's experiments, maybe the answer is "there's more redundancy than I expected," which would be interesting.) If there's a lot more intentional computation of XORs than I expected, then why? (I've found the speculation that LLMs might just computing a bunch of XORs up front because they don't know what they'll need later interesting.) I could just update my world model to "lots of XORs exist for either reasons (1) or (2)," but I sure would be interested in knowing which of (1) or (2) it is and why.

^{^}
I know that for CCS you're more worried about issues around correlations with features like true_according_to_Alice, but my feeling is that we might be able to handle spurious features that are that crazy and numerous, but not spurious features as crazy and numerous as these XORs.

[-]Rohin Shah2yΩ340

I think you mostly need to hope that it doesn't matter (because the crazy XOR directions aren't too salient) or come up with some new idea.

Yeah certainly I'd expect the crazy XOR directions aren't too salient.

I'll note that if it ends up these XOR directions don't matter for generalization in practice, then I start to feel better about CCS (along with other linear probing techniques). I know that for CCS you're more worried about issues around correlations with features like true_according_to_Alice, but my feeling is that we might be able to handle spurious features that are that crazy and numerous, but not spurious features as crazy and numerous as these XORs.

Imo "true according to Alice" is nowhere near as "crazy" a feature as "has_true XOR has_banana". It seems useful for the LLM to model what is true according to Alice! (Possibly I'm misunderstanding what you mean by "crazy" here.)

I'm not against linear probing techniques in general. I like linear probes, they seem like a very useful tool. I also like contrast pairs. But I would basically always use these techniques in a supervised way, because I don't see a great reason to expect unsupervised methods to work better.

If I had to articulate my reason for being surprised here, it'd be something like:
I didn't expect LLMs to compute many XORs incidentally
I didn't expect LLMs to compute many XORs because they are useful
but lots of XORs seem to get computed anyway.

This is reasonable. My disagreement is mostly that I think LLMs are complicated things and do lots of incidental stuff we don't yet understand. So I shouldn't feel too surprised by any given observation that could be explained by an incidental hypothesis. But idk it doesn't seem like an important point.

[-]Sam Marks2yΩ340

Imo "true according to Alice" is nowhere near as "crazy" a feature as "has_true XOR has_banana". It seems useful for the LLM to model what is true according to Alice! (Possibly I'm misunderstanding what you mean by "crazy" here.)

I agree with this! (And it's what I was trying to say; sorry if I was unclear.) My point is that
{ features which are as crazy as "true according to Alice" (i.e., not too crazy)}
seems potentially manageable, where as
{ features which are as crazy as arbitrary boolean functions of other features }
seems totally unmanageable.

Thanks, as always, for the thoughtful replies.

[-]ryan_greenblatt2yΩ690

Prediction: token level features or other extremely salient features are XOR'd with more things than less salient features. And if you find less salient things which are linearly represented, a bunch of this won't be XOR'd.

This solves the exponential blow up and should also make sense with your experimental results (where all of the features under consideration are probably in the top 50 or so for salience).

[-]Sam Marks2yΩ340

(Are you saying that you think factuality is one of the 50 most salient features when the model processes inputs like "The city of Chicago is not in Madagascar."? I think I'd be pretty surprised by this.)

(To be clear, factuality is one of the most salient feature relative to the cities/neg_cities datasets, but it seems like the right notion of salience here is relative to the full data distribution.)

[-]ryan_greenblatt2yΩ242

Yes, that's what I'm saying. I think this is right? Note that we only need salience on one side between false and true, so "true vs false" is salient as long as "false" is salient. I would guess that "this is false" is very salient for this type of data even for a normal pretrained LLM.

(Similarly, "this is english" isn't salient in a dataset of only english, but is salient in a dataset with both english and spanish: salience depends on variation. Really, the salient thing here is "this is spanish" or "this is false" and then the model will maybe XOR these with the other salient features. I think just doing the XOR on one "side" is sufficient for always being able to compute the XOR, but maybe I'm confused or thinking about this wrong.)

[-]Sam Marks2yΩ471

Idk, I think I would guess that all of the most salient features will be things related to the meaning of the statement at a more basic level. E.g. things like: the statement is finished (i.e. isn't an ongoing sentence), the statement is in English, the statement ends in a word which is the name of a country, etc.

My intuition here is mostly based on looking at lots of max activating dataset examples for SAE features for smaller models (many of which relate to basic semantic categories for words or to basic syntax), so it could be bad here (both because of model size and because the period token might carry more meta-level "summarized" information about the preceding statement).

Anyway, not really a crux, I would agree with you for some not-too-much-larger value of 50.

[-]DanielVarga2yΩ152

I’ll say that a model linearly represents a binary feature f if there is a linear probe out of the model’s latent space which is accurate for classifying f

If a model linearly represents features a and b, then it automatically linearly represents and $a \lor b$ .

I think I misunderstand your definition. Let feature a be represented by x_1 > 0.5, and let feature b be represented by x_2 > 0.5. Let x_i be iid uniform [0, 1]. Isn't that a counterexample to (a and b) being linearly representable?

[-]Sam Marks2yΩ120

Thanks, you're correct that my definition breaks in this case. I will say that this situation is a bit pathological for two reasons:

The mode of a uniform distribution doesn't coincide with its mean.
The variance of the multivariate uniform distribution is largest along the direction $x_{1} + x_{2}$ , which is exactly the direction which we would want to represent a AND b.

I'm not sure exactly which assumptions should be imposed to avoid pathologies like this, but maybe something of the form: we are working with boolean features $f$ whose class-conditional distributions $D (\cdot | f), D (\cdot | \neg f)$ satisfy properties like

$D (\cdot | f), D (\cdot | \neg f)$ are unimodal, and their modes coincide with their means
The variance of $D (\cdot | f), D (\cdot | \neg f)$ along any direction is not too large relative to the difference of the means $E_{x \sim D (\cdot | f)} (x) - E_{x \sim D (\cdot | \neg f)} (x)$

[-]Vivek Hebbar2yΩ230

The variance of the multivariate uniform distribution is largest along the direction $x_{1} + x_{2}$ , which is exactly the direction which we would want to represent a AND b.

The variance is actually the same in all directions. One can sanity-check by integration that the variance is 1/12 both along the axis and along the diagonal.

In fact, there's nothing special about the uniform distribution here: The variance should be independent of direction for any N-dimensional joint distribution where the N constituent distributions are independent and have equal variance.^[1]

The diagram in the post showing that "and" is linearly represented works if the features are represented discretely (so that there are exactly 4 points for 2 binary features, instead of a distribution for each combination). As soon as you start defining features with thresholds like DanielVarga did, the argument stops going through in general, and the claim can become false.

The stuff about unimodality doesn't seem relevant to me, and in fact seems directionally wrong.

^{^}
I have a not-fully-verbalized proof which I don't have time to write out

[-]Sam Marks2yΩ230

Thanks, you're totally right about the equal variance thing -- I had stupidly thought that the projection of onto y = x would be uniform on $[- \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}]$ (obviously false!).

The case of a fully discrete distribution (supported in this case on four points) seems like a very special case of a something more general, where a "more typical" special case would be something like:

if a, b are both false, then sample from $N (0, Σ)$
if a is true and b is false, then sample from $N (μ_{a}, Σ)$
if a is false and b is true then sample from $N (μ_{b}, Σ)$
if a and b are true, then sample from $N (μ_{a} + μ_{b}, Σ)$

for some $μ_{a}, μ_{b} \in R^{n}$ and covariance matrix $Σ$ . In general, I don't really expect the class-conditional distributions to be Gaussian, nor for the class-conditional covariances to be independent of the class. But I do expect something broadly like this, where the distributions are concentrated around their class-conditional means with probability falling off as you move further from the class-conditional mean (hence unimodality), and that the class-conditional variances are not too big relative to the distance between the clusters.

Given that longer explanation, does the unimodality thing still seem directionally wrong?

[-]Vivek Hebbar2yΩ110

Oops, I misunderstood what you meant by unimodality earlier. Your comment seems broadly correct now (except for the variance thing). I would still guess that unimodality isn't precisely the right well-behavedness desideratum, but I retract the "directionally wrong".

[-]Donald Hobson2yΩ240

Take this set contains exponentially many points. Is there Any function $f : {0, 1}^{n} \to R^{p o l y n}$ such that all exponentially many xor combos can be found by a linear probe?

This is a question of pure maths, it involves no neural networks. And I think it would be highly informative.

[-]Olli Järviniemi5mo40

There is no such function ; the output dimension needs to be at least $2^{n / 2} - 1$ for this to be possible.

Suppose that $f : {0, 1}^{n} \to R^{d}$ is such that $f (S)$ and $f (S^{c})$ are linearly separable for any XOR-subset (subset of form ${x \in {0, 1}^{n} : x_{i_{1}} \oplus x_{i_{2}} \oplus \dots \oplus x_{i_{k}} = 0}$ ). There are $2^{n}$ such XOR-subsets. Consider the matrix $M$ of dimension $2^{n} \times 2^{n}$ whose rows are labeled by $x \in {0, 1}^{n}$ and columns by XOR-subsets $S$ , with

$M_{x, S} = 1 if x \in S, else - 1.$

(I.e. $M$ is a Hadamard matrix of size $2^{n} \times 2^{n}$ . We may assume $M$ is symmetric.) The function $f$ is such that, for any $S$ , there exist $w_{S} \in R^{d}, b_{S} \in R$ such that

$sign (⟨ w_{S}, f (x) ⟩ + b_{S}) = M_{x, S}$

for all $x$ . Thus, if we define $v_{x} = (f (x), 1) \in R^{d + 1}$ and $u_{S} = (w_{S}, b_{S}) \in R^{d + 1}$ , we have

$M_{x, S} = sign (⟨ v_{x}, u_{S} ⟩)$ .

The definition of sign-rank of a matrix $M$ is the smallest dimension $d + 1$ for which such a decomposition exists. A theorem by Forster implies that the sign-rank of $M$ is at least $2^{n} / | | M | |$ , whereas it's well-known that the spectral norm of symmetric Hadamard matrices is $\sqrt{2^{n}}$ , That implies $d + 1 \geq 2^{n / 2}$ .

[-]Olli Järviniemi2y10

FWIW, I thought about this for an hour and didn't really get anywhere. This is totally not my field though, so I might be missing something easy or well-known.

My thoughts on the subject are below in case someone is interested.

Problem statement:

"Is it possible to embed the points ${0, 1}^{n}$ to $R^{poly (n)}$ (i.e. construct a function $f : {0, 1}^{n} \to R^{poly (n)}$ ) such that, for any XOR-subset $S$ of ${0, 1}^{n}$ , the set $f (S)$ and its complement $f (S^{c})$ are linearly separable (i.e. there is a hyperplane separating them)?"

Here XOR-subset refers to a set of form

$S = {x \in {0, 1}^{n} | x_{i_{1}} \oplus x_{i_{2}} \oplus \dots x_{i_{k}} = 0}$ ,

where $k$ is a positive integer and $i_{j} \in {1, \dots, n}$ are indices.

Observations I made:

One can embed ${0, 1}^{2}$ to $R ²$ in the desired way: take a triangle and a point from within it.
One cannot embed ${0, 1}^{n}$ to $R^{n - 1}$ .
- (Proof: consider the $n$ XOR-subsets where $k = 1$ . The corresponding hyperplanes divide $R^{n - 1}$ to at most $2^{n} - 1$ regions; hence some two points are embedded in the same region. This is a contradiction.)
If you have $m$ points in $R^{d}$ , only at most $m^{d + 1}$ of the exponentially many subsets of the $m$ points can be linearly separated from their complement.
- (Proof: If a subset can be separated, it can also be separated by a hyperplane parallel to a hyperplane formed by some $d$ of our points. This limits our hyperplanes to $(\frac{m}{d}) \approx m^{d}$ orientations. Any orientation gives you at most $m$ subsets.)
- This doesn't really help: the number of XOR-subsets is linear in the number of points. In other words, we are looking at only very few specific subsets, and general observations about all subsets are too crude.
XOR-subsets have the property "if $A$ and $B$ are XOR-subsets, then $(A \cap B) \cup (A^{c} \cap B^{c})$ is too" (the "XOR of XOR-subsets is a XOR-subset")
- This has a nice geometric interpretation: if you draw the separating hyperplanes, which divide the space into 4 regions, the points landing in the "diagonally opposite" regions form a XOR-subset.
- This feels like, informally, that you need to use a new dimension for the XOR, meaning that you need a lot of dimensions.

I currently lean towards the answer to the problem being "no", though I'm quite uncertain. (The connection to XORs of features is also a bit unclear to me, as I have the same confusion as DanielVarga in another comment.)

[-]Donald Hobson2y20

The connection to features is that if the answer is no, there is no possible way the network could have arbitrary X-or combos of features that are linearly represented. It must be only representing some small subset of them. (probably the xor's of 2 or 3 features, but not 100 features.)

Also, your maths description of the question matches what I was trying to express.

[-]Adam Newgas7mo30

I've come up with my own explanation for why this happens: https://www.lesswrong.com/posts/QpbdkECXAdLFThhGg/computational-superposition-in-a-toy-model-of-the-u-and#XOR_Circuits

In short, XOR representations are naturally learnt when a model is targetting some other boolean operation as the same circuitry makes all boolean operations linearly representable. But XOR requires different weights to identity operations, so linear probes still will tend to learn generalizable solutions.

[-]Sam Marks2yΩ130

If anyone would like to replicate these results, the code can be found in the rax branch of my geometry-of-truth repo. This was adapted from a codebase I used on a different project, so there's a lot of uneeded stuff in this repo. The important parts here are:

The datasets: cities_alice.csv and neg_cities_alice.csv (for the main experiments I describe), cities_distractor.csv and neg_cities_distractor.csv (for the experiments with banana/shed at the end of factual statements), and xor.csv (for the experiments with true/false and banana/shed after random text).
xor_probing.ipynb: my code for doing the probing and making the plots. This assumes that the activations have already been extracted and saved using generate_acts.py (see the readme for info about how to use generate_acts.py).

Unless you want to do PCA visualizations, I'd probably recommend just taking my datasets and quickly writing your own code to do the probing experiments, rather than spending time trying to figure out my infrastructure here.

[-]Gabe M2yΩ230

Suppose has a natural interpretation as a feature that the model would want to track and do downstream computation with, e.g. if a = “first name is Michael” and b = “last name is Jordan” then $a \land b$ can be naturally interpreted as “is Michael Jordan”. In this case, it wouldn’t be surprising the model computed this AND as $f (x) = R e L U ((v_{a} + v_{b}) \cdot x + b_{\land})$ and stored the result along some direction $v_{f}$ independent of $v_{a}$ and $v_{b}$ . Assuming the model has done this, we could then linearly extract $a \oplus b$ with the probe
$p_{a \oplus b} (x) = σ (- (α v_{f} + v_{a} + v_{b}) \cdot x + b_{\oplus})$
for some appropriate $α > 1$ and $b_{\oplus}$ .^[7]

Should the $-$ be inside the inner parentheses, like $σ ((- α v_{f} + v_{a} + v_{b}) \cdot x + b_{\oplus})$ for $α > 1$ ?

In the original equation, if $a$ AND $b$ are both present in $x$ , the vectors $v_{a}$ , $v_{b}$ , and $v_{f}$ would all contribute to a positive inner product with $x$ , assuming $α > 1$ . However, for XOR we want the $v_{a}$ and $v_{b}$ inner products to be opposing the $v_{f}$ inner product such that we can flip the sign inside the sigmoid in the $a$ AND $b$ case, right?

[-]Sam Marks2yΩ120

Yes, you are correct, thanks. I'll edit the post when I get a chance.

[-]Lucius Bushnaq2y30

Maybe a⊕b is represented “indicentally” because NN representations are high-dimensional with lots of stuff represented by chance

This would be my first guess, conditioned on the observation being real, except strike “by chance”. The model likely wants to form representations that can serve to solve a very wide class of prediction tasks over the data with very few non-linearities used, ideally none, as in a linear probe. That’s pretty much the hallmark of a good general representation you can use for many tasks.

I thus don't think that comparing to a model with randomized weights is a good falsification. I wouldn’t expect a randomly initialized model to have nice general representations.

My stated hypothesis here would then predict that the linear probes for XOR features get progressively worse if you apply them to earlier layers. Because the model hasn't had time to make the representation as general that early in the computation. So accuracy should start to drop as you look at layers before fourteen.

I'll also say that if you can figure out a pattern in how particular directions get used as components for many different boolean classification tasks, that seems like the kind of thing that might result in an increased understanding of what these directions encode exactly. What does the layer representation contain, in actual practice, that allows it to do this?

[-]ryan_greenblatt2y*Ω120

I think that linearly available XOR would occur if the model makes linearly available any boolean function which is "linearly independent" from the two values individually. So, maybe this could be implemented via something other than XOR, which is maybe more natural?

[-]Sam Marks2yΩ121

What is "this"? It sounds like you're gesturing at the same thing I discuss in the section "Maybe is represented “incidentally” because it’s possible to aggregate noisy signals from many features which are correlated with boolean functions of a and b"

[-]Sam Marks2yΩ242

The thing that remains confusing here is that for arbitrary features like these, it's not obvious why the model is computing any nontrivial boolean function of them and storing it along a different direction. And if the answer is "the model computes this boolean function of arbitrary features" then the downstream consequences are the same, I think.

[-]ryan_greenblatt2yΩ130

I edited my comment. I'm just trying to say that like how you get for free, you also get XOR for free if you compute anything else which is "linearly independent" frrom the components a and b. (For a slightly fuzzy notion of linear independence where we just need separability.)

[-]Charlie Steiner2yΩ120

Wild.

The difference in variability doesn't seem like it's enough to explain the generalization, if your PC-axed plots are on the same scale. But maybe that's misleading because the datapoints are still kinda muddled in the has_alice xor has_not plot, and separating them might require going to more dimensions, that have smaller variability.

[-]Review Bot2y*10

The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

[-]Review Bot2y*10

[-]Vivek Hebbar2yΩ110

Here's a fun thing I noticed:

There are 16 boolean functions of two variables. Now consider an embedding that maps each of the four pairs {(A=true, B=true), (A=true, B=false), ...} to a point in 2d space. For any such embedding, at most 14 of the 16 functions will be representable with a linear decision boundary.

For the "default" embedding (x=A, y=B), xor and its complement are the two excluded functions. If we rearrange the points such that xor is linearly represented, we always lose some other function (and its complement). In fact, there are 7 meaningfully distinct colinearity-free embeddings, each of which excludes a different pair of functions.^[1]

I wonder how this situation scales for higher dimensions and variable counts. It would also make sense to consider sparse features (which allow superposition to get good average performance).

^{^}
The one unexcludable pair is ("always true", "always false").
These are the seven embeddings:

[-]Tao Lin2y10

I'd be interested in experiments with more diverse data. Maybe this only works because the passages are very short and simple and uniform, and are using very superposition-y information that wouldn't exist in longer and more diverse text

[-]Vivek Hebbar2yΩ110

Maybe models track which features are basic and enforce that these features be more salient

Couldn't it just write derivative features more weakly, and therefore not need any tracking mechanism other than the magnitude itself?

[-]Sam Marks2yΩ120

Some features which are computed from other features should probably themselves be treated as basic and thus represented with large salience.

[-]c.trout2y10

My apologies if this is dumb but if when a model linearly represents features a and b it automatically linearly represents and $a \lor b$ , then why wouldn't it automatically (i.e. without using up more model capacity) also linearly represent $a \oplus b$ ? After all, $a \oplus b$ is equivalent to $(a \lor b) \land \neg (a \land b)$ , which is equivalent to $(a \lor b) \land (\neg a \lor \neg b)$ .

In general, since { $\neg$ , $\land$ } is truth functionally complete, if a and b are represented linearly won't the model have a linear representation of every expression of first order logic (without quantifiers) that involve only a and b?

(If I'm missing something dumb, feel free to ignore this post).

[-]ryan_greenblatt2y*80

I think the key thing is that and $a \lor b$ aren't separately linearly represented (or really even linearly represented in some strong sense). The model "represents" these by $a + b$ with different thresholds like this:

So, if we try to compute $a \lor b \land \neg (a \land b)$ linearly then we would do $a \lor b$ corresponds to $(a + b)$ , $a \land b$ corresponds to $(a + b)$ and $\neg$ corresponds to negation, so we get $(a + b) - (a + b)$ which clearly doesn't do what we want!

If we could apply a relu on one side, then we could get this: $(a + b) - 2 r e l u ((a + b) - 1)$ . And this could work (assuming $a$ and $b$ are booleans which are either 0 or 1). See also Fabien's comment which shows that you naturally get xor just using random MLPs assuming some duplication.

[-]ryan_greenblatt2y20

More generally, just try to draw the XOR decision boundary on the above diagram!

[-]c.trout2y10

Ah, I see now! Thanks for the clarifications!

[-]Gabe M2y10

If true, this would be a big deal: if we could figure out how the model is distinguishing between basic feature directions and other directions, we might be able to use that to find all of the basic feature directions.

Or conversely, and maybe more importantly for interp, we could use this to find the less basic, more complex features. Possibly that would form a better definition for "concepts" if this is possible.

[-]Matteo Capucci2y10

Here's what my spidey sense is telling me: model is trying to fit as many representation as possible (IIRC this is known in mech interp) and by mere pushing apart features $a$, $b$ in a large dimensional space you end up making $a \oplus b$ linearly separable. That is, there might be a combinatorial phenomenon underlying this, which feels counterintuitive because of the large dimensions involved.

[-]Daniel Kokotajlo2yΩ211

indicentally

typo

^{^}

To be clear, this is not a direct quote, and Rohin explicitly clarified that he didn’t expect this to be true for arbitrary features a and b. Rohin only claimed that this was true in the case they were studying, and that he would guess “taking XORs of features” is a common motif in NNs.

^{^}

E.g. suppose the model is unaware of some feature f, but does have a direction corresponding to some feature f’ which is perfectly correlated with f in our data. According to the definition I use in this post, the model linearly represents f; this is not the way I would usually use the term.

^{^}

Throughout, I’ll always draw directions as if they’re orthogonal directions in the model’s latent space. It’s indeed the case that the model might represent features in superposition, so that these directions are not orthogonal, or even linearly independent. But that doesn’t change the basic dynamic: that the model must allocate additional capacity in order to represent the feature $a \oplus b$ .

^{^}

Chosen to be the same hidden state as in my Geometry of Truth paper.

^{^}

When taking into account superposition among features, there are subtle geometrical issues one needs to worry about as well, which I discuss in section 4.1 of my truth paper.

^{^}

Another option would have been to just reinitialize the weights according to some distribution. Resetting the network in this way is a bit more principled for experiments of this sort, because it erases everything the model learned during training, but maintains lots of the basic statistical properties of the NN weights.

^{^}

The nonlinearity in the computation of f is essential for this to work.

^{^}

This above-and-beyond is needed for the same reason that the nonlinearity above was needed.

Statement	has_alice	has_not	label
Bob: The city of Kaifeng is in Mexico.	F	F	F
Bob: The city of Calgary is in Canada.	F	F	T
Bob: The city of Chicago is not in the United States.	F	T	F
Bob: The city of Shanghai is not in Mexico.	F	T	T
Alice: The city of Tijuana is in the United Arab Emirates.	T	F	F
Alice: The city of Bangkok is in Thailand.	T	F	T
Alice: The city of Turin is not in Italy.	T	T	F
Alice: The city of Oslo is not in Venezuela.	T	T	T

LESSWRONG
LW

LESSWRONG
LW

159

What’s up with LLMs representing XORs of arbitrary features?

159

Ω 75

159

Ω 75

Maybe models track which features are basic and enforce that these features be more salient

The RAX claim: LLMs linearly represent XORs of arbitrary features, even when there’s no reason to do so

RAX would be very surprising

Without additional assumptions, RAX implies linear probes shouldn’t generalize

Models have exponentially more stuff than you thought they did

The evidence: RAX is true in every case I’ve checked

What does RAX mean for model internals research?

Qualitatively new reasons for linear probes not to generalize

Results of probing experiments are much harder to interpret

Applications of interpretability need to either have a way to distinguish XORs of features from basic features, or need to be robust to an exponential increase in number of features

What’s going on?

Basic features are more salient than XORs

Maybe $a \oplus b$ is represented “incidentally” because NN representations are high-dimensional with lots of stuff represented by chance

Maybe $a \oplus b$ is represented “incidentally” because it’s possible to aggregate noisy signals from many features which are correlated with boolean functions of a and b

Maybe models track which features are basic and enforce that these features be more salient

Models compute a bunch, but not all, XORs in a way that we don’t currently understand

159

What’s up with LLMs representing XORs of arbitrary features?

159

Ω 75

159

Ω 75

Maybe models track which features are basic and enforce that these features be more salient

The RAX claim: LLMs linearly represent XORs of arbitrary features, even when there’s no reason to do so

RAX would be very surprising

Without additional assumptions, RAX implies linear probes shouldn’t generalize

Models have exponentially more stuff than you thought they did

The evidence: RAX is true in every case I’ve checked

What does RAX mean for model internals research?

Qualitatively new reasons for linear probes not to generalize

Results of probing experiments are much harder to interpret

Applications of interpretability need to either have a way to distinguish XORs of features from basic features, or need to be robust to an exponential increase in number of features

What’s going on?

Basic features are more salient than XORs

Maybe a⊕b is represented “incidentally” because NN representations are high-dimensional with lots of stuff represented by chance

Maybe a⊕b is represented “incidentally” because it’s possible to aggregate noisy signals from many features which are correlated with boolean functions of a and b

Maybe models track which features are basic and enforce that these features be more salient

Models compute a bunch, but not all, XORs in a way that we don’t currently understand

Maybe $a \oplus b$ is represented “incidentally” because NN representations are high-dimensional with lots of stuff represented by chance

Maybe $a \oplus b$ is represented “incidentally” because it’s possible to aggregate noisy signals from many features which are correlated with boolean functions of a and b