There are at least two distinct ways to conceptually organize a scientific endeavor: top-down or bottom-up. The bottom-up approach involves finding the smallest objects we can confidently understand, studying them, and leveraging that knowledge to study more complex objects made up of those smaller pieces. As a highly simplified example, in mech-interp this might mean starting by first understanding single neurons/SAE directions/features, leveraging this understanding to study circuits/manifolds composed of those neurons, and finally building a understanding of the model as a composition of circuits. By way of analogy, if our goal was restore a photo, the bottom-up approach starts by restoring the individual pixels, using this as a guide to restore patches, and finally to put it all together.

The top-down approach instead tries to get a coarse picture of the whole object, and uses those intuitions to understand motifs, and then zooms into mechanics of those motifs. For the mech-interp, this might involve getting some vague sense of what sort of information the model contains the what the model does with that information, studying the subspaces and broad phases that create and manage that information, and then finally study the mechanics of those pieces. Following the previous analogy, we start by getting a blurry version of the whole picture, and use this to identify the most important broad pieces to study, and finally extend the understanding the whole model.

This post contains some initial attempts on doing top-down interpretability. These are really just a lightly edited version of my own research notes, and are largely exploratory and preliminary. Most of these are conclusions drawn from a wide variety of particular experiments and tests-- if you have interest in any particular part I can add a comment where I write up the evidence in more detail.

The post is roughly organized around the following broad curiosity:
How is the residual stream organized? What does it contain? How is it used by the model?

As a final note before the main content: I study Gemma-4-31B-base here. I chose this model fairly arbitrarily, primarily motived by the belief that this model is sufficiently intelligent to be worth studying, the architecture is simple (there is no MoE), and I can run it locally. I have not done a meaningful comparison to other models, but a cursory look suggests the model's scape is far more anisotropic than others. This ended up being helpful, so I stuck with this model.

TL;DR: I looked at how the residual stream of Gemma-4-31B-base is organized, starting from its coarse shape. I find the residual stream highly anisotropic and shift a lot across layers in raw coordinates, but becomes more balanced and stable across layers when we read the stream through the pre-MLP RMSNorm. I study this 'corrected' space and find that the middle layers (~L20–L40) split into a few distinct parts. I zoom into one of them, a set of ~600 small and unusually stable coordinates that seems to function as a crucial reference for the model and the common source of most of its computations. Keeping only these coordinates (plus a handful of norm-controlling ones) at the final position nearly preserves the model’s predictions, while keeping the other ~90% breaks it. Patching experiments suggest this space holds the current token along with its contextual meaning (e.g. which sense of “bank” is meant). This is my first interpretability project and these are lightly edited research notes, so treat the conclusions as preliminary. I mainly wanted some technical practice to up skill into the field, and wanted to share the results in case they are useful for anyone else.

What is the Residual Stream's Shape?

I'll start with a few observations about Gemma-4-31B in particular. Gemma-4-31B uses a sandwich norm, which means the RMSNorm is applied both before an operation, and after.

Because the RMSNorm is applied both before and after either the Attention or MLP operation, the attention/mlp operations are somewhat buffered from need to 'internally' use the same scale as the residual stream. The model can, somewhat freely, retain its own internal coordinate scale while storing a different one in the residual stream.

The residual stream in Gemma-4-31B is, as a result, highly anisotropic. I averaged the residual stream over ~1024 passages (right before the MLP block) to produce this table.

which shows that the large coordinates (95% percentile) are about ~2 OOMs larger than the small coordinates (5% percentile).

The residual stream was really never meant to be measured in raw coordinates, and Gemma's extreme unevenness is nice illustration of this fact. The more 'natural' coordinates for the residual stream is the size after applying the RMSNorm. I'll shelve the term that comes from dividing the size of the residual stream for now, and focus just on the pre-MLP diagonal .

A reasonable first hypothesis is that is roughly equally sized in all directions, but only in its 'natural' coordinate system (As mentioned, we will shelve the term for now, and focus just on the diagonal). What would this mean? Assume a coordinate in the residual stream varies between and , and coordinate varies between and . If the model viewed these coordinates as equally sized, then the diagonal for should be , and the diagonal for should be . This way, in the 'natural' metric, both coordinates vary between and . If this is the case, coordinate that are naturally smaller in raw size within the residual are compensated by larger diagonal, and vice-versa for naturally larger coordinates.

We can test each coordinates 'natural variance' by computing the value of the residual stream across some examples and measuring the size of one standard deviation in that coordinate. At layer 20, this gives us the following (log-log) plot:

Perhaps unsurprisingly, the diagonal gain per coordinate is almost perfectly inversely proportional to the standard deviation across nearly all orders of magnitudes (i.e. ). That is, small coordinates really are read with a large diagonal to compensate for being small, and large coordinate are read with a small diagonal to compensate for being large.

Here are the values from fitting a line with slope (in log space) onto the various layers SD to diagonal relationship.

Thus, the first thing we learn is that the model seems to 'work' in a metric where the coordinate activations are roughly isotropic, even though the raw coordinates are not. The of the part of the RMSNorm is also quite stable and, at each layer, is roughly constant (for reason I'll address in the next section). I'll denote the median value of by (I've dropped the subscript of the layer for convenience, but does vary per layer). Since for each layer, this gives us what seems to be a reasonable norm to work in. So, from this point on, I'll largely not use raw residual stream coordinates as the norm-- instead it seems that when we measure we ought to first correct by multiplying .

Once we switch to this new coordinate system , the residual stream starts to look a lot more stable than it did before. Whereas before the residual stream varied over multiple orders of magnitude, is quite stable. We'll denote the residual stream after the diagonal mapping as (or for the ith coordinate), and write when we want to emphasize the layer.

Here is a plot of the residual stream, averaged over 128 passages, and then transformed into the new coordinate system. Each dot has as its x coordinate the size of one standard deviation in that coordinate and as its y coordinate the average value of (i.e. the coordinates are ), Despite coordinates varying over many orders of magnitude in size, the coordinates stay within a fairly narrow range in the new coordinate system.

For the sake of the next two graphs, I add one more constant parameter per layer to make the average height 1.

In this graph, I color the coordinates by how much, on average, they move between layers. This gives a sense of which coordinates are actively varying, and which are staying stable. More precisely, we color by.

There are various interesting patterns to track in this graph, which we will look at in more detail later. For instance, there seems to be a stable space at a small scale, a much more active space in the larger scale, and a surprising amount of stability between each update, and large transformations at various specific layers. As a final check, we can apply this to a random single token at a random sentence to check that we are stabilizing the scale on particular examples as well.

We see that the diagonal is normalizing the scale here, and about the same motifs as the averaged example (two distinct sizes, spaces that are stable, multiple phases of computation).

A few observations to follow and explore in the next section:

• The residual stream has a anisotropic shape, but becomes isotropic once we look at the residual stream as the model uses it.
• The residual stream appears to bifurcate into two main scales. Plotting on a linear scale, most of the coordinates bunch at once scale, with a smaller group clumping at a smaller scale. There is additionally a third, small group whose size is too large to fit into the graph.

What are the Parts and Phases of the Residual Stream?

The residual stream contains multiple distinct phases and parts. The majority of my interest and research has been in the middle phase, which I define as roughly starting at roughly L20, and ending at L40. I will say more on the phases later, but let us first look at study this middle region and identify its parts.

As mentioned in the last section, the residual stream bifurcates into two distinct main scales, with a third small clump of large coordinates. Let us first look at that third small clump.

Scale 1: Very large Coordinates

If we plot the entire natural variation versus norm gain, we find that there are a few noticeable outliers.

These coordinates have both very large size, and very low diagonal readout strength. That means these coordinates are large in the raw residual stream, but very small to the actual attention computation. That is, they contribute strongly to the norm computation, but not much to the actual MLP computation. The coordinate 2041 at layer 25 alone contributes ~ of the norm energy, but only ~ of the readout energy. Here is a plot of a few of the coordinates across the middle layers, and their norm contribution.

These coordinates contribute most of the norm size, but are read out with a very small size. Removing these tokens effect on the norm has a major impact on loss, whereas zeroing out there read does practically nothing.

I believe these norm control coordinates have been observed many times, so I will mainly comment on the importance for this post.

• When we make interventions, we should leave the norm coordinates alone. For instance, when we zero, we should ensure we don't zero the norm coordinates. Zeroing the norm coordinates messes with the scale the model like to work in.
• We must be especially careful in cases where we adjust for scale by inverting the diagonal . (For a case where we might do this: if we wanted to study the Jacobian matrix, then Gemma's unbalanced raw residual norm would make small coordinates unreasonably influential and large coordinates unreasonably ineffective (per unit norm change). Changing a coordinate that varies between and by appears to have a much larger effect, but this may only be because our perturbation is not in the right metric). Since these coordinates have such a small read strength, is huge in these coordinates. However, these coordinates have a small read strength for a separate reason then other coordinates do-- the model intends to ignore these coordinates in its computations. In most cases, I treat the norm coordinates separately than the rest, and just hold them out of most computations. The model seems to want to keep these directions roughly constant, and so I try to respect this preference in interventions and computations.

I will refer to these coordinates as the Norm coordinates. I conservatively pick out 53 of these coordinates that are outliers within the L20-L40 range, and so also refer to these as the Norm53 coordinates. Though just 3 coordinates is basically sufficient to capture all the norm energy.

Scale 2: Very Small Coordinates

Most of the remainder of this post will be focused on these nature of the small coordinates. There is a set of roughly ~600 coordinates that have a conspicuously smaller size than the rest of the coordinates between the layers L20-L40. I'll refer to these as the Quiet coordinates or Quiet600, since these coordinates are 'quiet' within the residual stream. It is worth noting that this is a bit of a misnomer though, since these are perhaps the most important coordinates within the residual stream.

First, let us get a birds eye view of 'activity' within each coordinate. We use the diagonal to suggest the natural 'size' of the coordinate, and color each coordinate by how large the writes to each coordinate are. For instance, the diagonal was , then the natural range of the coordinate is viewed to be to , and so a write of would account for of the coordinates range, so would be colored in the green-blue range. A write of would be of the implied size and be colored in the orange-red range. I've split the Quiet600 into two parts, the Persistent279, and the rest of the Quiet coordinates. There are a few other spaces I highlight as outliers, though I might not get to them until the next post. Also note that I follow the same coordinate across layers, so one row follows a single fixed coordinate from layer to layer .

There are at least a few important observations.

The large 'Other' set is nearly uniform, which is evidence that it is not coordinate aligned. If the space were not coordinate aligned, then a single coordinate is essentially a random probe into its state, so when we average over many examples every coordinate looks identical. Thus, the majority of the residual stream is not coordinate aligned.

The Quiet600 space stands out. This suggests these coordinates are a part of some subspace that is coordinate aligned. This doesn't imply the individual coordinates are meaningful, but that there is a distinctive subspace contained within those 600 coordinates.

The writes to Quiet600 are much smaller than the rest of the space. These writes are much smaller than the writes elsewhere.

The same set of coordinates remain distinctive throughout the middle phase. For the Quiet remain coordinates, they seem to remain stable from ~L20 to ~L35, whereas the 'Persistent' set remains stable until the start of ~L40. This means that this distinctive space is staying within within the same subspace.

There are a few other outlier groups (Late377, Late171), but the Quiet set apparently captures nearly all the coordinate aligned subspace.

This is enough reason to be interested in this space, let us now understand it.

How is the Quiet Space distinct?

The Quiet (and its subset the Persistent) space have a few properties that make then distinctive.

First, this space is highly stable. The stability of this space very high, but this fact can easily be obscured by measuring in raw coordinates. At each layer, there is a constant that multiplies the whole residual stream, so any stable space is forced to decay across layers. Indeed, this set of coordinates does undergo that decay in size, which you may have observed in the earlier gif as one set of coordinates seemed to move moving leftward.

Mathematically, across L20 - L40 any stable data ought to decay purely from the layer-wise scaling. Thus, the data in L40 ought to be about ~x smaller than it was at the start of L20. And indeed, the data in the 'Persistent' set does decay, we empirically see the median standard deviation decrease by x (that this is less than x suggest some small writes that are aligned with the current coordinates, and therefore slow the decay), while the median reader in these coordinates increases by ~x. In raw residual coordinates this space is undergoing a significant change in scale, but after adjusting for the MLP read the median reader is nearly constant in scale at ~x.

When we measure each layer's residual stream in the MLP coordinates , we find that the persistent set has a high cosine similarity between adjacent layers, and drifts only very gradually.

Should be view it as surprising that there are some coordinates that remain stable for many layers? Given that each layer does have a fully dense write, some coordinates should make it through a few blocks simply by chance. Heuristically, if we take the writes at each layer (and adjust to SD) and then redistribute those to random coordinates we can get a sense of how many coordinates should slip through on an example just by luck.

In general, many coordinates survive a few blocks simply because they haven't had time to be overwritten, but we expect less than one to survive past 4 blocks. And this is to say nothing about the same coordinates surviving multiple examples. The persistent of these coordinates is not expected if writes did not intentionally organize to protect these coordinates.Indeed, this is exactly what we find.

Second, this space is protected from writes. Not only are writes small on average, but this space is protected from large writes by the weights themselves. The post-MLP RMSNorm has diagonals that are much smaller than the coordinate natural size.

There is an easy way to test whether the data in these coordinates is really stably carried forward by the model. We take their value at L20, and transplant them at L35 (changing only the scale). Let's do this just to the persistent coordinates and measure the loss. I test this on a relatively long context of 512 tokens, applying this transplant to all tokens between 16 and 511, and measure the loss change at equally sized increments. We find that change has only a very small effect on the loss. Of course, it is also possible that the loss impact is small because the model doesn't use this space much. So we test against replacing the coordinates with their mean, or replacing their value with the coordinates at another position, and either of these choices pretty much cook the model.

We can test, more specifically, what happens if we hold the value in the persistent space stable (while accounting for its changing scale). We take the deviation against the mean , and before each MLP block we replace the persistent part of the residual stream with , where is a fitted scalar per layer to adjust the global scale, and is the average within the persistent coordinates (computed on separate documents to avoid leakage). This method gives a loss increase of ~ mnat when applied between L20-L35. Thus, the persistent coordinates are stable enough that simply carrying forward the data from L20 has only a relatively small effect on the loss.

Third, this space is crucial to the model computation, and arguably the most central part of its next-token prediction. For these experiments, I take a much larger set of documents (~3000), so that I can be pretty confident in the loss impact being approximately right.

To motivate some of these experiment, I should note that a hypothesis I have about the early L0-L20 phase, is that its main roles is to construct the data in this quiet/persistent space (and maybe a bit of something about organizing how attention functions). If one believes this hypothesis, at around the L20 basically none of the rest of the residual stream matters and, when it does matter, it is only about attention. So, for these experiments I am making interventions only at the final token whose job is to predict the next token, the previous tokens hidden states are unchanged.

Indeed, we find that the model has surprisingly little difference when we remove everything outside the Quiet space at around L20. Replacing every other coordinate (except the norm) keeps only about ~12% of the residual stream, but nearly perfectly matches the original model's predictions.

At H22, it costs only ~13 mnats of loss, and roughly of this comes from redistributing probability when the leading token stays unchanged (the leading token matches in of cases).

Replacing coordinates by their means still can carry a lot of information-- nonetheless even zeroing out every other coordinate has a similarly low loss impact.

KL divergence is also within the same range, at ~ mnats for the zero replacement and ~ mnats for the mean replacement. Since KL divergence is higher, this suggests some of the added variation is actually helpful to the loss, though of course loss still increases so most of it is not.

How does this compare against other baselines? Retaining only 600 random coordinates largely destroys the model, and retaining 2,500 coordinates gives something similar.

These next measurements are on a much smaller set (I chose to save some compute by not reducing error bars on the clearly large values), so the actual loss values are not comparable to the ones above, but give a general sense of size.

Even keeping the entire complement of the quiet set (~90% of coordinates) and the norm coordinates leaves the model broken. More than any other coordinates, the model needs the quiet coordinates to function.

What does this teach us about the space. It gives us some pretty strong hypotheses about what it contains

• It is created in the early phase of computation. Its too early to encode the answer, and the model seems to put in real effort to maintaining the space. It is protected, and remains somewhat close to constant. Importantly, keeping hundreds of coordinates nearly constant comes at a meaningful cost to the residual stream size. We are basically sacrificing of the size. This suggests that these coordinates ought to be referenced and used often. If they were carried for no reason, the loss ought to reclaim them to make the residual stream larger again.
• These coordinates are crucial to the loss. They seem to contain the information most crucial to running the computation. They coordinates alone are sufficient to organize the full future computation at the current position with high fidelity.
• This space is always there, and always in the same place. The fact that the model places the content in the same space would seem to suggest that many aspects of the models computation start at the quiet space. It seems plausible that parts of the model uses this space to get its 'bearings' and figure out how to start a computation.

What do the Quiet Coordinates Contain?

Let us now turn out attention to studying what's carried in these coordinates. I will save studying the more broad functional role of these coordinates for another post (with a few initial thoughts towards the end of this section), and focus instead on mainly what is in these coordinates.

Give the hypotheses in the last section, it seemed likely to me that the quiet space carries some enriched version of what the current token is. This is the kind thing you might expect the model to create early and reference often. I have studied mainly the persistent set (which I'll label as ), so that is the part of the quiet set I'll look at in this post.

The easiest way I can see to confidently check what a given space contains is through patching. Whatever the model can tell us about when we patch it in must actually be in . We'll initially use two different ways to readout what contains: and . The methodology is displayed below.

We get our patch by just saying 'here's a word' and we extract the activations in the persistent set. The important difference between these two is that the patch functions entirely through the MLP. The patch applies to the same position that produces the answer-- there is no gap in position between the patch and the output. This measures whether the model can use the patch through only the same position MLP. The case has a gap between the patch and the answer, so it necessarily must be using the information in the patch through the attention mechanism. In all cases, I use the word thing as the token I patch into. This way, the word wolf (or whatever other word) never appears in the context. If the model can properly write wolf, it is only because contains the information necessary to encode the wolf.

Unsurprisingly, the MLP version basically works for any word. We can take a word, donate it into L20, and the model will copy it.

The attention version requires an initial fix before it works. By default, after the patch, the model will just give the definition of thing writing A word used to indicate a person or thing previously mentioned or easily identified.. This suggests to me that the consequences of the word being thing has already propagated out of this space into the 'Other' coordinates. If we suppress the write at MLP1, then the model will define the word we've patched in. Here are a few examples:

For a word like bat there is more than one possible definition. Does the definition it gives depend on the context? We might wonder whether the patch contains the contextual meaning of word. We can test this by applying under patches that have contextually changed the meaning of a word. Again, we only patch at the word bat, so whatever is different we get must come from attention bringing this context into the Protected set at that position.

Indeed, the does differentiate between the meaning in the different contexts, both in explict cases like the one above and in implicit cases.

The model is perfect at this task. It occasionally makes mistakes, especially on definitions that are closely related. I plan to study this more closely in the next post-- I would expect that the model does delineate between these cases, but perhaps there are a few coordinates I am missing that is causing it be occasionally wrong.

We can also test if models can distinguish identical words across languages (false friends). pain in English means 'pain', but 'bread' in French. Indeed it does. Using this prompt: The language of this word is {language}. Here is the word: {word}, the model can often distinguish between the same token in different languages

We obtain similar results if we construct more natural sentences (such as The anaesthetic wore off during the night, and I woke up in pain vs Chaque matin, je passe à la boulangerie pour acheter du pain.

At this point, we have enough preliminary data to form some meaningful hypotheses and start to do some more interesting tests. So far, it seems:

• The space carries the token itself. Patching is sufficient to obtain the token and its meaning
• encodes the meaning of a word and the contextual meaning. The model pulls together nearby context into this space in order to produce a rich understanding of the token.

We might additionally wonder the degree to which the model actually looks to this space for properties about a word. It certainly looks here for the meaning and the contextual usage of the word, but what else does it read?

More on Quiet Coordinates

These next section is more preliminary, but I include it because it gives some sense of the kind of role that has, and what sort of things are possible to do with the space beyond changing the token.

As a initial test, we can see what kind of steering effects we can produce by editing . Unsurprisingly, there are directions to change the plurality and various other linguistic features of a word.

More interestingly, we can also change the meaning of a word without changing how its copied. For instance, adding boy-girl, queen-king, plus a few other pairs gives a roughly male-female direction within . Adding this to the word aunt gives but .

This suggests that the model largely 'trusts' the data in the protected coordinates. Even when it can still identify that the word is aunt, it still trusts the wrong traits we've inserted into that words definition. We can test this by injecting in a number of words together and seeing how the object is defined.

The model remains calling this a wolf but its definition now combines features of each of the animals. It is a carnivorous mammal of the dog family that has a long snount, bushy tail, and typically yellow-brown fur.

For this next part, I haven't done the real organized analysis all the way yet, so I'll just share some of the interesting things we can find in this space. I use a zoo of different methods to extract the vectors, I can give more details in the next post when I've organized the work a bit more. The gist is that I use singular values of the J-lens matrix to extract the strongest coherent steering vectors, alongside singular values of the J-lens gram matrix (so instead of averaging , I average ) to get the most influential steering vectors.

Among these influential steering directions are many that change the models interpretation of what sort of document it is in. For instance, SC-0010's a direction which makes the model interpret the document its in as a business material. For instance (prompt in italics),

Mara set a plain ceramic mug on the kitchen table. She had ten minutes before leaving for work. As she picked it up, she thought of how much better it would be if the mug were made by her favorite local pottery shop. The shop used high-quality clay and employed talented artists. The mug’s value went beyond the materials used.

If you are a business owner, you know that it’s not just about the product...

On another prompt, adding SC-0010 again (somewhat gracefully) veers the story to a business context.

Event notes: Mara and Ivo met in the public garden on Saturday. They moved six fallen branches off a path, then sat on a bench for ten minutes.

The following paragraph describes this event, keeping the people and facts the same.
Mara's diary entry:
I met Ivo in the park on Saturday. I charged him 30€ to remove six logs from his path. I got 40% of the payment upfront, with the rest due in a month.

There are various other directions that effect how the model frames the initial prompt. We can get some sense of this variety by some of the different ways a model can construe the same prompt.

Prompt: She had not been back in eleven years. At the ticket counter, she put down her suitcase and said, “One way, please. I’m going home to

Baseline: stay.” He stared at her. “One way, ma’am? You’re sure you’re not making a mistake?” “I’m positive.” She smiled, and the gray eyes behind the glasses were clear. “I’m going home.”

Here are a few of the different framings adding a steering vector to can produce
(Framed as personal commentary, +4SD Gram-3)

die."

When I first read this in the summer of 2002, I read the story over and over again, as if by doing so I would be able to extract a little more of the author's wisdom. It was a story about a woman who, in the first sentence, we discover is about to die, and in the final sentence, we are told that she has died.
It was a story about a woman who is going home to die.
I have not read it since. I do not need to re-read it, for I have not forgotten it. I have only to think of the story, and I am filled with the profound grief of the old woman, who, in her dying, has only one wish, that is, to

(Reader's commentary, +8 SD, Gram-20)

stay." He stared at her. "One way, ma'am? You're sure you won't be coming back?" "Yes," she said. "I'm sure."

I love this story. The narrator is the ticket agent. And the story is written in such a way that we the readers do not know anything about the woman other than what the narrator tells us. We don't know where she is going, why she hasn't been there for eleven years, why she is returning, and why she is returning now. And I think that is the beauty of this story. It can be applied to all of us. We can insert our own thoughts into the story to make it ours.

For example, the woman could be going to the town she...

(Author's bio)

stay." And so, the journey begins, as it does for millions of Americans who are trying to reconcile their personal dreams with the needs of their families. But this particular journey takes a dramatic twist when the protagonist, a woman of middle age, finds herself in a small-town church with the same minister she had not seen in twenty-four years. The sparks are still there. But the obstacles are formidable. Will the embers of their long-ago love blaze again? Or will it be too little, too late?

Author Bio:
John Richard, an American, has lived in France, Germany, and the Philippines. He is now retired and lives in Texas with his wife. They have two adult sons.
This is his third novel. His first, The Price of Peace,

Some directions can capture surprisingly abstract aspects of the framing. SC-0022 at L30 is particularly interesting. Its J-lens readout is
+ </u>, </strong>, beautiful, flowers, flower, river, floral, lilies, mermaid, waterfalls, watercolor, meadows, waterfall, pastel, реки (rivers), изображения (images), rainbow, trees, sculpture, sunrise, decorative, Beautiful, beauty, фотографии (photographs)

- funding, negoti, policies, lobbying, criticizing, advocating, ,“, Negoti, financing, negotiators, disagreed, advisers, ,…, Ethics, Critics, criticized, seeking, enforcing, financed, ,”, willing, regarding, agents, Funding.

With ~ of its energy in the J-lens mean direction, meaning the direction generally pushes in a direction aligned with the + and - directions, but still most of its influence comes from a contextual component. As is the case for many directions, the readout is best read as a second order influence: how is the model changing its stance or orientation such that those words would be likely to said? In this case, the top words are come from visual depecition and scenery, whereas the minus directions are abstract

Thus, an initial label is pastoral, visual-focused depiction mode versus an evaluative mode.

Prompt: On an ordinary Tuesday, a person arrived at a place they knew well. This time, they noticed

Baseline: something new.
The new feature was a small notice board that had been installed near the entrance. The person approached it, curiosity piqued. The notice board contained a detailed report about a project. It was interesting, but the person had seen similar reports before.
However, this report was different. It was...

(-8 SD)

something amiss. They sought the guidance of their trusted local attorney and were introduced to the services of an experienced lawyer, who, after a thorough evaluation, determined that the individual had indeed been wronged. With the attorney's backing, the person sought a court order to have their rights upheld. The judge, upon reviewing the...

(+8 SD)

something different. A special flower caught their eye. It was like a rainbow in a garden. They felt something special about it, like a quiet hello from a friend.

As they walked closer, they saw a girl playing. She was holding the flower. She turned and smiled, her eyes bright like stars.

In the ticket counter example, with (+4 SD)

She had not been back in eleven years. At the ticket counter, she put down her suitcase and said, "One way, please. I'm going home to see my mother." She looked at the face in the mirror and could not recognize it. Her eyes were dull, her skin was gray, and her hair was a tangle of dirty blond and brown. She wondered if her mother would recognize her, but her mother was blind.

The bus lurched into the yard, and there was her mother, standing by the lilac bush, her face like a bright yellow apple, her white hair blowing in the wind, the wind blowing through her.

"Ma?"

"Oh, my baby. I knew you'd come...

Which is excessively full of descriptive language. Compar to (-4 SD)

"Ma'am, do you know the war is still going on there?"

"I know. I'm going anyway."

"I'm sorry, ma'am. We can't sell you a ticket to go there."

"You can't stop me."

"Please, ma'am. There's a war going on. It's very dangerous."

"I know. I just have to go."

"We can't sell you a ticket."

She went to the counter at the...

is full of directions like these which change the model's construal of the context its read, often in interesting and abstract ways. The results are too tentative so far to say anything too concrete, but it seems to be the case that the model stores a kind of annotated understanding of the document within this space.

These direction seem especially salient in cases where there is ambiguity in the future generation. For instance, many of the steering effects don't happen until a new line, where the model likely has to decide what 'kind' of document its in. For instance: was the start a preamble to something new, or is the beginning of a longer story, or what kind of objects are relevant to talk about?

I'll end by giving my current best understanding of the space so far. There are fairly large error bars here, but I think these are useful hypotheses.

• The quiet/persistent set functions like an enriched token, with a large number of useful and linearly readable properties. The early layers first build up this quiet space. It starts with low level details, like the part of speech, whether it is a fragment of another word, surface-level linguistic details, etc. It then brings in contextual data to construct the meaning of the word. Most of this building is done by ~L20.
• Information still trickles into the quiet set during L20-L39. These are probably increasingly abstract and broad annotations related to the token, such as what kind of document its a part of.
• Much of the information in the residual stream takes the following route. First, it seems that of the 'operations' the model has access to are functions of this enriched token space. For instance, in a graph traversal task, the operations to traverse from one node to another are binded to tokens.
• It seems that the model looks at the persistent set plus some other coordinates, decides what to do, and then posts like a 'job request' into the other coordinates. When the 'job request' hits the MLP/Attention block that can fulfill it, it is then operated on. Often I would guess the result is a new 'job request' with new data stored also in the 'Other' coordinates. I view this kind of 'job request' framing likely because we generally have so much leeway in switching around the order of blocks or skipping blocks. This seems plausibly to be a useful organization, you shove into each layer a large number of computation and context-gated functions that run on request. This 'job request' phase stops at L40.
• At L40, the model stops its abstract computational phase, and moves into some new phase. I do not have much insight into this phase-- except that I would expect its more dense and operates on more of the space at once. L40 seems the cutoff for whatever jobs are being done-- in some experiments you can use a readout to see that the model is still making progress looping through a computation but even still the same major L40 shift happens. So the model can't request more time in the computation phase if its not done, it just gets cut off.

Most of this section is speculation and guesswork, but they form hypotheses and starting points to investigate for the next section. And in any case, this is exactly how top-down interp ought to go-- we start with a blurry estimation of the model behavior, de-noise until we can find the parts that are right and wrong, and then try another updated blurry estimation.