Can we define something's function in terms of the selection pressure it's going to face in the future?

Consider the case where the part of my brain that lights up for horses lights up for a cow at night. Do I have a broken horse detector or a working horse-or-cow-at-night detector?

I don't really see how the past-selection-pressure account of function can be used to say that this brain region is malfunctioning when it lights up for cows at night. Lighting up for cows at night might never have been selected against (let's assume no-one in my evolutionary history has ever seen a cow at night). And lighting up for this particular cow at night has been selected for just as much as lighting up for any particular horse has (i.e. it's come about as a side effect of the fact that it usefully lit up for some different animals in the past).

But maybe the future-selection-pressure account could handle this? Suppose the fact that the region lit up for this cow causes me to die young, so future people become a bit more likely to have it just light up for horses. Or suppose it causes me to tell my friend I saw a horse, and he says, "That was a cow; you can tell from the horns," and from then on it stops lighting up for cows at night. In either of these cases we can say that the behaviour of that brain region was going to be selected against, and so it was malfunctioning.

Whereas if lighting up for a cows-at-night as well as horses is selected for going forward, then I'm happy to say it's a horse-or-cow-at-night detector even if it causes me to call a cow a horse sometimes.

I appreciate this write-up! I felt like the answer to "is the truth of a statement represented somewhere" is obviously no (elaboration below). However I was surprised by the "tell a story about purpose / natural fact angle". It seems that a malfunctioning heart is "wrong" in a different, more obvious, sense than e.g. a false mathematical statement.

Here are the points that I thought were obvious (and this still makes sense to me):

• If you measure correlation between real things (horse at night) and NN activations (looks like a horse at night) there will be a mismatch.
  • This feels like a kind of "skill issue" to me: your class of things (which included "horse at night" but not "looks like a horse at night") wasn't sufficiently wide or good. We should just recognise that say the examples correlated with this direction are all consistent with the NN thinking something is a horse, and then labelling it correctly. Obviously this is hard (impossible? especially of the NN is smarter then us), but the issue doesn't seem that the NN made a mistake per se. Instead the issue was choosing only real physical things as possible labels.
• It's hard to define "misrepresentation" as a physical phenomenon based on the physical atoms present.
  • This seems somewhat obvious to me, of course there is no representation of whether someone is "right" or "wrong" in the atoms! That'd be cheating! Imagine writing "The Riemann hypothesis is true" or "The runtime of busy beaver (99) is an even number". The compute by the universe when you write this is less than it takes to decide whether (at least the second) hypothesis is correct or not!

However, at the end of the post you suggest there might actually be a way to get to a definition of misrepresentation or malfunction based on physical atoms only:
The (correct / intended) representation is that which makes a more natural (less surprising?) story. The heart having the purpose of pumping blood explains its design a lot better than it having the purpose of making a thump-thump sound, even when all you have is a malfunctioning heart.

I think this is a valuable read for people who work in interp but feel like I want to add a few ideas:

• Distinguishing Misrepresentation from Mismeasurement: Interpretability researchers use techniques that find vectors which we say correspond to the representations of the model, but the methods we use to find those may be imperfect. For example, if your cat SAE feature also lights up on racoons, then maybe this is a true property of the model's cat detector (that is also lights up on racoons) or maybe this is an artefact of the SAE loss function. Maybe the true cat detector doesn't get fooled by racoons, but your SAE latent is biased in some way. See this paper that I supervised for more concrete observations.
• What are the canonical units? It may be that there is a real sense in which the model has a cat detector but maybe at the layer at which you tried to detect it, the cat detector is imperfect. If the model doesn't function as if it has an imperfect cat detector then maybe downstream of the cat-detector is some circuitry for catching/correcting specific errors. This means that finding the local cat detector you've found which might have misrepresentation issues isn't in itself sufficient to argue that the model as a whole has those issues. Selection pressures apply to the network as a whole and not necessarily always to the components. The fact that we see so much modularity is probably not random (John's written about this) but if I'm not mistaken, we don't have strong reasons to believe that the thing that looks like a cat detector must be the model's one true cat detector. 

I'd be excited for some empirical work following up on this. One idea might be to train toy models which are incentivised to contain imperfect detectors (eg; there is a noisy signal but reward is optimised by having a bias toward recall or precision in some of the intermediate inferences). Identifying intermediate representations in such models could be interesting. 

Off the top of my head, not very well-structured:

It seems the core thing we want our models to handle here is the concept of approximation errors, no? The "horse" symbol has mutual information with the approximation of a horse; the Santa Claus feature existing corresponds to something approximately like Santa Claus existing. The approach of "features are chiseled into the model/mind by an imperfect optimization process to fulfil specific functions" is then one way to start tackling this approximation problem. But it kind of just punts all the difficult parts onto "how the optimization landscape looks like".

Namely: the needed notion of approximation is pretty tricky to define. What are the labels of the dimensions of the space in which errors are made? What is the "topological picture" of these errors?

We'd usually formalize it as something like "this feature activates on all images within $ϵ$ MSE distance of horse-containing images". And indeed, that seems to work well for the "horse vs cow-at-night" confusion.

But consider Santa Claus. That feature "denotes" a physical entity. Yet, what it {responds to}/{is formed because of} are not actual physical entities that are approximately similar to Santa Claus, or look like Santa Claus. Rather, it's a sociocultural phenomenon, which produces sociocultural messaging patterns that are pretty similar to sociocultural messaging patterns which would've been generated if Santa Claus existed^[1].

If we consider a child fooled into believing into Santa Claus, what actually happened there is something like:

• "What adults tell you about the physical world" is usually correlated with what actually exists in the physical world.
• The child learns a model that maps adults' signaling patterns onto world-model states.
• The world-states "Santa Claus exists" and "there's a grand adult conspiracy to fool you into believing that Santa Claus exists" correspond to fairly similar adult signaling patterns.
  • At this step, we potentially are again working with something neat like "vectors encoding adults' signaling patterns", and the notion of similarity is e. g. cosine similarity between those vectors.
• If the child's approximation error is significant enough to fail to distinguish between those signaling patterns, they pick the feature to learn based on e. g. the simplicity prior, and get fooled.

Going further, consider ghosts. Imagine ghost hunters equipped with a bunch of paranormal-investigation tools. They do some investigating and conclude that their readings are consistent with "there's a ghost". The issue isn't merely that there's such a small distance between "there's a ghost" and "there's no ghost" tool-output-vectors that the former fall within the approximation error of the latter. The issue is that the ghost hunters learned a completely incorrect model in which some tool-outputs which don't, in reality, correspond to ghosts existing, are mapped to ghosts existing.

Which, in turn, presumably happened because they'd previously confused the sociocultural messaging pattern of "tons of people are fooled into thinking these tools work" with "these tools work".

Which sheds some further light at the Santa Claus example too. Our sociocultural messaging about Santa Claus is not actually similar to the messaging in the counterfactual where Santa Claus really existed^[2]. It's only similar in the deeply incomplete children's models of how those messaging patterns work...

Summing up, I think a merely correlational definition can still be made to work, as long as you:

• Assume that feature vectors activate in response to approximations of their referents.
• Assume that the approximation errors can lie in learned abstract encodings ("ghost-hunting tools' outputs", "sociocultural messaging patterns"), not only in default encodings ("token embeddings of words"), with likewise-learned custom similarity metrics.
• Assume that learned abstract encodings can themselves be incorrect, due to approximation errors in previously learned encodings, in a deeply compounding way...
  • ... such that some features don't end up "approximately corresponding" to any actually existing phenomenon. (Like, the ground-truth prediction error between similar-sounding models is unbounded in the general case: approximately similar-sounding models don't produce approximately similarly correct predictions. Or even predictions that live in the same sample space.)

... Or something like that.

1. ^{^}

   Well, not quite, but bear with me.

2. ^{^}

   E. g., there'd be more fear about the eldritch magical entity manipulating our children.

I think that representation is best explained as both correspondence and the outcome of optimization - specifically, representation is some sort of correspondence (which can be loose) that is caused by some sort of optimization process. 

I'll speak primarily in defense of correspondence since I think that is where we disagree. 

"All models are wrong, but some are useful" is a common aphorism in statistics, and I think it is helpful here too. You seem to treat mistaken representations as a separate sort of representation. However, even an ordinarily correct representation can contain some mistaken elements. For example:

• A child might correctly identify a horse, but not know that horses are made out of atoms.
• A man in 1970 might correctly identify a campfire, but mistakenly believe that the fire is a release of phlogiston. 

Likewise ordinarily mistaken representations can contain correct elements:

• If I think I've seen a horse when I've actually seen a cow at night, I've still correctly identified a thing-which-looks-like-a-horse. I may still correctly predict that others will see it as a horse too.
• A child who believes in Santa Claus will be likely to correctly predict the behavior of a "mall Santa" i.e. an actor pretending to be Santa in a public place. 

There are also edge cases where a representation mixes correct and incorrect elements, such that it isn't clear whether we should call it a mistaken representation or not:

• Methyl methacrylate adhesive is often sold under the name "plastic epoxy." Chemically speaking, it's not an epoxy at all - but practically speaking it is a binary adhesive that is used just like an epoxy. If you think that it is epoxy, you will only be wrong in ways that (almost) never matter.
• If you think that a tomato is a vegetable, then you are conventionally considered correct in a culinary context. However, by the botanical definition, a tomato is actually a fruit.

This suggests that it is useful to stop thinking about mistaken and correct representations as separate types, but rather to think about representations having mistaken and correct elements. 

Having made this shift, I think that the correspondence theory of representation becomes viable again. Even a representation that is conventionally classified as mistaken may contain many correct elements - enough correct elements to make it about whatever it is about. A child's representation of Santa Claus contains many correct elements (often wears red, jolly, brings presents) and one very prominent incorrect element (the child thinks that Santa physically exists rather than being a well-known fiction). It may very well be the case that most of the bits in the child's representation are correct; we just pay more attention to the few that are wrong. For another example: if I think that I see a horse, but I actually see a cow at night, the correct elements include "it looks like a horse to me," and "it's that thing over there that I'm looking at right now." There's a lot of specificity in that last correct element! I think that's enough to make my representation be about the cow. 

On the other hand, we can consider examples where an optimization process exists but where it fails to create correspondence:

• A child tells you that Santa lives in deserts, has a venomous stinger, and is a close relative of spiders. You'd probably say that they aren't actually talking about Santa at all. They're describing a scorpion and getting the name wrong.
• Once in high school, when assigned to write an essay about a historical figure, I accidentally wrote my first draft about a different figure with the same name. If my teacher was mean, they could have interpreted this first draft as being about the first figure and docked me many points becasue I had nearly every fact about them wrong. Instead, they noticed the real mistake (which was in a sense their mistake; the assignment was ambiguous) and marked accordingly. 

With that being said, I do agree with you that optimization is an important piece of the puzzle - but not becasue it can explain how something can be about something else even if it is mistaken. Rather, I think that optimization is the answer to the problem of coincidences. For example:

• A novelist writes a murder-mystery which, by coincidence, is a correct description of a real murder. The naive correspondence theory of representation says that the novel is about the real murder; common sense says that it is not. 

Adding the second criteria - that the correspondence must be caused by an optimization process - prevents a definition of representation from identifying coincidences as representations. 

the heart has been optimized (by evolution) to pump blood; that’s a sense in which its purpose is to pump blood.

Should we expect any components like that inside neural networks?

Is there any optimization pressure on any particular subcomponent? You can have perfect object recognition using components like "horse or night cow or submarine or a back leg of a tarantula" provided that there is enough of them and they are neatly arranged.

I think this is missing the mechanics of interpretability. Interpretability is about the opposite, "what it does"

So basically, interpretability only cares about mixed features (malfunction where the thing is not as labeled) only insofar as the feature does not only do the thing that the label would make us think that it does.

That is to say, in addition to labeling representational parts of the model, interp wants to know the relation between those parts 

So we know ultimatley enough about what the model will do to either debug as capabilities research, or prove that it will not try to do x, y, and z that will kill us for safety. 

Basically, for an alignment researcher the polysemantics that come from being wrong sometimes, if the wrongness really is in the model, so produces the same actions, that is basically okay.

Even just plain polysemantics is not the end of the world for you interp tools, because there is not one "right" semantics. You just want to span the model behaviour.