tl;dr: I reproduced the Goodfire lab's cyclical manifold result using days-of-the-week on Gemma-2-2b and found that Anthropic's pre-trained CLT features produce an even cleaner cyclical manifold with more distinct features than the raw activations or output probabilities. Colab notebook implementation can be found here.
Introduction
In May of this year, Wurgaft et al. (2026) reproduced a known result that days-of-the-week (and other semantically cyclical and sequential relationships) are represented on a cyclical manifold in activation space (Engels et al. 2024). They went on to uncover an isometric manifold in what they term "behavior space", which is a Hellinger projection of the model's token output probabilities, and to show that steering the model along this manifold produces predictable outcomes (i.e. the generated output moves through the days of the week in order as the model is shifted).
I was inspired by this result, along with a recent revisit of Ameisen et al.'s Circuit Tracing work from Anthropic, to explore whether this isometry was also present in cross-layer transcoder (CLT) features. The preliminary exploration described below demonstrates that this cyclical manifold is, in fact, present in the pre-trained CLT features of Gemma-2-2b. My results also suggest that the feature extraction process is working exactly as designed, producing more distinct day-of-the-week features with cleaner and simpler semantic relationships than either the raw activations or model output probabilities.
Methods
Wurgaft et al. (2026) ran their experiments at layer 28 (of 32) of Llama 3.1 8b, with the explanation that using a late layer would ensure that concept geometries are fully computed. They don't specify, but I assume that they did not use any of the final layers since activations in those layers tend to be more focused on the token production than the semantic content (Li and Subramani, 2025). My experiments used layer 20 (of 26) Gemma-2-2b as a roughly equivalent layer position in a model that Anthropic has released pre-trained CLT features for.
I used the same day-of-the-week input prompt format from Wurgraft et al., using "Q: What day is k days after z? \n A:" to prompt the model to produce a certain day token (and to enforce the cyclical relationships between the days). The residual stream and logits were then collected for the last token of the prompt to capture information about the predicted next token[1]. The residual stream activations, , were used raw, and the output logits, , were run through a Softmax layer before undergoing conversion into Hellinger space (a simple square root, as shown below) for geometric interpretation.
The CLT features were computed through simple matrix multiplication as shown below, where is the residual stream, is the CLT encoder, and is the CLT bias term for layer 20:
I followed Wurgaft et al.'s methods for fitting cubic splines through the extracted feature centroids, essentially looking to uncover a cyclical relationship between the average outputs for each of the predicted day-of-the-week representations.
Results and Discussion
As seen in the 2D PCA projection above, all three spaces produce a clearly cyclical manifold for the day-of-the-week representations, with no days that overlap or appear out of order. Out of the three projections, it is clear that the CLT features produce the cleanest circle in two dimensions, with the individual feature vectors visually more clustered around each centroid.
This cosine similarity heatmap also demonstrates that the CLT features are much more cleanly distinct and separated in CLT feature space than the representations in either the activations or behavior space. The high values in activation space are not particularly surprising, since cosine similarity is frequently inflated and a poor descriptor of distributional geometry in high dimensions. CLT feature space avoids this, with each day-of-the-week centroid cleanly separated and gradually decreasing and then increasing in similarity to the other centroids as we traverse the 7-day cycle (i.e. Thursday is minimally similar to Monday, Friday is minimally similar to Tuesday, etc.). This is exactly what the CLT feature extraction method was designed to do, taking the raw residual stream and producing cleaner, more interpretable semantic features.
Normalized Arc Length between Day-of-the-Week Centroids
Mon
Tues
Wed
Thurs
Fri
Sat
Sun
Activation
0.113
0.159
0.176
0.254
0.046
0.101
0.151
CLT
0.144
0.280
0.185
0.112
0.104
0.088
0.117
Behavior
0.097
0.123
0.136
0.182
0.143
0.137
0.182
Considering the geodesic distances (the normalized arc lengths between day-of-the-week centroids shown in the table above), we actually see that the behavior space has the most regularly spaced centroids, the activation space is notably more irregular, and the CLT feature space falls somewhere in between. The irregularity in the activation space, where the weekend centroids are separated from the mid-week centroids, may be explained by research from cognitive science (Ellis et al. 2015) which shows that weekend days are processed differently than weekdays. This same research may still explain some of the irregularity in the CLT manifold, though it's less pronounced, and the weekend/weekday split is less visible in the geodesic distances.
Conclusion and Future Work
The results in this preliminary exploration largely serve to support the concepts behind the CLT feature extraction process. The same semantic relationships evident in the raw activations are present in the CLT feature space, but in a cleaner, more distinct, and more interpretable format. The fact that this cyclical manifold does exist suggests that the steering methods from Wurgraft et al. (2026) are also likely to carry over, though I've yet to complete that experiment in full.
Assuming that manifold steering is effective in CLT feature space, this opens the door to discovery and manipulation of other semantic or behavioral manifolds, using the more interpretable CLT features to identify relationships that may be better represented by a geometric manifold than a linear direction (as is currently most commonly used in steering work).
I actually ran my first experiment by using prompts like "It is [time of day] on [day of week]." and collected the activations and logits over the day-of-the-week token that was produced. Somewhat surprisingly, the results still showed a cyclical manifold (even in the behavioral space). However, the interpretation for the "representative" manifold was not as clear as that of a "predictive" manifold (especially when looking at the output logits), and there would be little to no expectation of steerability along that manifold.
tl;dr: I reproduced the Goodfire lab's cyclical manifold result using days-of-the-week on Gemma-2-2b and found that Anthropic's pre-trained CLT features produce an even cleaner cyclical manifold with more distinct features than the raw activations or output probabilities. Colab notebook implementation can be found here.
Introduction
In May of this year, Wurgaft et al. (2026) reproduced a known result that days-of-the-week (and other semantically cyclical and sequential relationships) are represented on a cyclical manifold in activation space (Engels et al. 2024). They went on to uncover an isometric manifold in what they term "behavior space", which is a Hellinger projection of the model's token output probabilities, and to show that steering the model along this manifold produces predictable outcomes (i.e. the generated output moves through the days of the week in order as the model is shifted).
I was inspired by this result, along with a recent revisit of Ameisen et al.'s Circuit Tracing work from Anthropic, to explore whether this isometry was also present in cross-layer transcoder (CLT) features. The preliminary exploration described below demonstrates that this cyclical manifold is, in fact, present in the pre-trained CLT features of Gemma-2-2b. My results also suggest that the feature extraction process is working exactly as designed, producing more distinct day-of-the-week features with cleaner and simpler semantic relationships than either the raw activations or model output probabilities.
Methods
Wurgaft et al. (2026) ran their experiments at layer 28 (of 32) of Llama 3.1 8b, with the explanation that using a late layer would ensure that concept geometries are fully computed. They don't specify, but I assume that they did not use any of the final layers since activations in those layers tend to be more focused on the token production than the semantic content (Li and Subramani, 2025). My experiments used layer 20 (of 26) Gemma-2-2b as a roughly equivalent layer position in a model that Anthropic has released pre-trained CLT features for.
I used the same day-of-the-week input prompt format from Wurgraft et al., using "Q: What day is k days after z? \n A:" to prompt the model to produce a certain day token (and to enforce the cyclical relationships between the days). The residual stream and logits were then collected for the last token of the prompt to capture information about the predicted next token[1]. The residual stream activations,
The CLT features were computed through simple matrix multiplication as shown below, where
I followed Wurgaft et al.'s methods for fitting cubic splines through the extracted feature centroids, essentially looking to uncover a cyclical relationship between the average outputs for each of the predicted day-of-the-week representations.
Results and Discussion
As seen in the 2D PCA projection above, all three spaces produce a clearly cyclical manifold for the day-of-the-week representations, with no days that overlap or appear out of order. Out of the three projections, it is clear that the CLT features produce the cleanest circle in two dimensions, with the individual feature vectors visually more clustered around each centroid.
This cosine similarity heatmap also demonstrates that the CLT features are much more cleanly distinct and separated in CLT feature space than the representations in either the activations or behavior space. The high values in activation space are not particularly surprising, since cosine similarity is frequently inflated and a poor descriptor of distributional geometry in high dimensions. CLT feature space avoids this, with each day-of-the-week centroid cleanly separated and gradually decreasing and then increasing in similarity to the other centroids as we traverse the 7-day cycle (i.e. Thursday is minimally similar to Monday, Friday is minimally similar to Tuesday, etc.). This is exactly what the CLT feature extraction method was designed to do, taking the raw residual stream and producing cleaner, more interpretable semantic features.
Normalized Arc Length between Day-of-the-Week Centroids
Mon
Tues
Wed
Thurs
Fri
Sat
Sun
Activation
0.113
0.159
0.176
0.254
0.046
0.101
0.151
CLT
0.144
0.280
0.185
0.112
0.104
0.088
0.117
Behavior
0.097
0.123
0.136
0.182
0.143
0.137
0.182
Considering the geodesic distances (the normalized arc lengths between day-of-the-week centroids shown in the table above), we actually see that the behavior space has the most regularly spaced centroids, the activation space is notably more irregular, and the CLT feature space falls somewhere in between. The irregularity in the activation space, where the weekend centroids are separated from the mid-week centroids, may be explained by research from cognitive science (Ellis et al. 2015) which shows that weekend days are processed differently than weekdays. This same research may still explain some of the irregularity in the CLT manifold, though it's less pronounced, and the weekend/weekday split is less visible in the geodesic distances.
Conclusion and Future Work
The results in this preliminary exploration largely serve to support the concepts behind the CLT feature extraction process. The same semantic relationships evident in the raw activations are present in the CLT feature space, but in a cleaner, more distinct, and more interpretable format. The fact that this cyclical manifold does exist suggests that the steering methods from Wurgraft et al. (2026) are also likely to carry over, though I've yet to complete that experiment in full.
Assuming that manifold steering is effective in CLT feature space, this opens the door to discovery and manipulation of other semantic or behavioral manifolds, using the more interpretable CLT features to identify relationships that may be better represented by a geometric manifold than a linear direction (as is currently most commonly used in steering work).
I actually ran my first experiment by using prompts like "It is [time of day] on [day of week]." and collected the activations and logits over the day-of-the-week token that was produced. Somewhat surprisingly, the results still showed a cyclical manifold (even in the behavioral space). However, the interpretation for the "representative" manifold was not as clear as that of a "predictive" manifold (especially when looking at the output logits), and there would be little to no expectation of steerability along that manifold.