tl;dr: I reproduced the Goodfire lab's cyclical manifold result using days-of-the-week on Gemma-2-2b and found that Anthropic's pre-trained CLT features produce an even cleaner cyclical manifold with more distinct features than the raw activations or output probabilities. Colab notebook implementation can be found here.
Introduction
In May of this year, Wurgaft et al. (2026) reproduced a known result that days-of-the-week (and other semantically cyclical and sequential relationships) are represented on a cyclical manifold in activation space (Engels et al. 2024). They went on to uncover an isometric manifold in what they term "behavior space", which is a Hellinger projection of the model's token output probabilities, and to show that steering the model along this manifold produces predictable outcomes (i.e. the generated output moves through the days of the week in order as the model is shifted).
I was inspired by this result, along with a recent revisit of Ameisen et al.'s Circuit Tracing work from Anthropic, to explore whether this isometry was also present in cross-layer transcoder (CLT) features. The preliminary exploration described below demonstrates that this cyclical manifold is, in fact, present in the pre-trained CLT features of Gemma-2-2b. My results also suggest that the feature extraction process is working exactly as designed, producing more distinct day-of-the-week features with cleaner and simpler semantic relationships than either the raw activations or model output probabilities.
Methods
Wurgaft et al. (2026) ran their experiments at layer 28 (of 32) of Llama 3.1 8b, with the explanation that using a late layer would ensure that concept geometries are fully computed. They don't specify, but I assume that they did not use any of the final layers since activations in those layers tend to be more focused on the token production than the semantic content (Li and Subramani, 2025). My experiments used layer 20 (of 26) Gemma-2-2b as a roughly equivalent layer position in a model that Anthropic has released pre-trained CLT features for.
I used the same day-of-the-week input prompt format from Wurgraft et al., using "Q: What day is k days after z? \n A:" to prompt the model to produce a certain day token (and to enforce the cyclical relationships between the days). The residual stream and logits were then collected for the last token of the prompt to capture information about the predicted next token[1]. The residual stream activations, , were used raw, and the output logits, , were run through a Softmax layer before undergoing conversion into Hellinger space (a simple square root, as shown below) for geometric interpretation.
The CLT features were computed through simple matrix multiplication as shown below, where is the residual stream, is the CLT encoder, and is the CLT bias term for layer 20:
I followed Wurgaft et al.'s methods for fitting cubic splines through the extracted feature centroids, essentially looking to uncover a cyclical relationship between the average outputs for each of the predicted day-of-the-week representations.
Results and Discussion
As seen in the 2D PCA projection above, all three spaces produce a clearly cyclical manifold for the day-of-the-week representations, with no days that overlap or appear out of order. Out of the three projections, it is clear that the CLT features produce the cleanest circle in two dimensions, with the individual feature vectors visually more clustered around each centroid.
This cosine similarity heatmap also demonstrates that the CLT features are much more cleanly distinct and separated in CLT feature space than the representations in either the activations or behavior space[2]. The high values in activation space are not particularly surprising, since cosine similarity is frequently inflated and a poor descriptor of distributional geometry in high dimensions. CLT feature space avoids this, with each day-of-the-week centroid cleanly separated and gradually decreasing and then increasing in similarity to the other centroids as we traverse the 7-day cycle (i.e. Thursday is minimally similar to Monday, Friday is minimally similar to Tuesday, etc.). This is exactly what the CLT feature extraction method was designed to do, taking the raw residual stream and producing cleaner, more interpretable semantic features.
Normalized Arc Length between Day-of-the-Week Centroids
Mon
Tues
Wed
Thurs
Fri
Sat
Sun
Activation
0.113
0.159
0.176
0.254
0.046
0.101
0.151
CLT
0.144
0.280
0.185
0.112
0.104
0.088
0.117
Behavior
0.097
0.123
0.136
0.182
0.143
0.137
0.182
Considering the geodesic distances (the normalized arc lengths between day-of-the-week centroids shown in the table above), we actually see that the behavior space has the most regularly spaced centroids, the activation space is notably more irregular, and the CLT feature space falls somewhere in between. The irregularity in the activation space, where the weekend centroids are separated from the mid-week centroids, may be explained by research from cognitive science (Ellis et al. 2015) which shows that weekend days are processed differently than weekdays. This same research may still explain some of the irregularity in the CLT manifold, though it's less pronounced, and the weekend/weekday split is less visible in the geodesic distances.
Conclusion and Future Work
The results in this preliminary exploration largely serve to support the concepts behind the CLT feature extraction process. The same semantic relationships evident in the raw activations are present in the CLT feature space, but in a cleaner, more distinct, and more interpretable format. The fact that this cyclical manifold does exist suggests that the steering methods from Wurgraft et al. (2026) are also likely to carry over, though I've yet to complete that experiment in full.
Assuming that manifold steering is effective in CLT feature space, this opens the door to discovery and manipulation of other semantic or behavioral manifolds, using the more interpretable CLT features to identify relationships that may be better represented by a geometric manifold than a linear direction (as is currently most commonly used in steering work).
Update
Emmanuel Ameisen, from Anthropic's Interpretability team, noted that CLT features can rely on the bias term to remove a positive mean direction, so it is more reasonable to compare the CLT features with the demeaned (or mean-centered) residuals. The mean-centered cosine similarity heatmaps below demonstrate how pairwise relationships can easily be masked by a non-zero mean.
In the activation space in particular, the transformation from near-perfect cosine similarities across-the-board to only very weak similarity with neighboring days and much stronger dissimilarity (negative cosine) with maximally distant days is remarkable, though it is explained by the same distributional characteristics. With a non-zero mean, this tightly-clustered distribution is pointing in a single direction (the direction of that non-zero mean), producing very high cosine similarities. However, once the distribution is moved to the origin, all of the points are very close to zero, causing even a very small difference in one dimension to be likely to push it across the origin and bring down the overall cosine similarity.
On these new plots, the distinction between the CLT feature space and the raw activation and behavior spaces becomes much less obvious, though the CLT feature space still shows the cleanest and most evenly spaced cyclical pattern out of the three. In the CLT space, all neighboring days show positive cosine similarities and all non-neighboring days show negative cosine similarities, while in the activation and behavior spaces the (relatively) high similarities are much less cleanly associated with direct neighbors.
In the activation and behavior spaces, the weekend grouping from the Ellis et al. (2015) cognitive science citation is much more apparent in the mean-centered plots, with higher similarities across Fri-Sat-Sun. Interestingly, this doesn't hold in the CLT space, where Fri-Sat and Sat-Sun do have slightly higher cosine similarities than other neighboring days, but Fri-Sun shows the same negative cosine similarity seen in other non-neighboring days--it would seem that the CLT space's strict adjacency structure outweighs any semantic grouping within the three weekend days. It's worth noting that the behavior space has some other unexpected groupings, such as the high similarity between Mon and Wed, that deserve more detailed exploration as well.
This update complicates the original result, as it shows that some of the cleanliness of the CLT feature space was an artifact of comparing against non-zero-mean distributions. Once centered, the distinction of the CLT feature space is in producing a perfectly clean neighboring-day-similarity structure, rather than as a large gap in overall similarity patterns. This is still a real finding, but much more modest than the original framing suggested, and a good reminder to check for shared-mean artifacts when using cosine similarity in high dimensions. Thanks again to Emmanuel Ameisen for the catch.
AI Use Disclosure
Claude Code was used heavily in developing the methods in the Colab notebook associated with this post and was consulted when interpreting results and designing figures for analysis. All writing is the author's own work, and all results, analyses, and conclusions were verified by the author.
I actually ran my first experiment by using prompts like "It is [time of day] on [day of week]." and collected the activations and logits over the day-of-the-week token that was produced. Somewhat surprisingly, the results still showed a cyclical manifold (even in the behavioral space). However, the interpretation for the "representative" manifold was not as clear as that of a "predictive" manifold (especially when looking at the output logits), and there would be little to no expectation of steerability along that manifold.
tl;dr: I reproduced the Goodfire lab's cyclical manifold result using days-of-the-week on Gemma-2-2b and found that Anthropic's pre-trained CLT features produce an even cleaner cyclical manifold with more distinct features than the raw activations or output probabilities. Colab notebook implementation can be found here.
Introduction
In May of this year, Wurgaft et al. (2026) reproduced a known result that days-of-the-week (and other semantically cyclical and sequential relationships) are represented on a cyclical manifold in activation space (Engels et al. 2024). They went on to uncover an isometric manifold in what they term "behavior space", which is a Hellinger projection of the model's token output probabilities, and to show that steering the model along this manifold produces predictable outcomes (i.e. the generated output moves through the days of the week in order as the model is shifted).
I was inspired by this result, along with a recent revisit of Ameisen et al.'s Circuit Tracing work from Anthropic, to explore whether this isometry was also present in cross-layer transcoder (CLT) features. The preliminary exploration described below demonstrates that this cyclical manifold is, in fact, present in the pre-trained CLT features of Gemma-2-2b. My results also suggest that the feature extraction process is working exactly as designed, producing more distinct day-of-the-week features with cleaner and simpler semantic relationships than either the raw activations or model output probabilities.
Methods
Wurgaft et al. (2026) ran their experiments at layer 28 (of 32) of Llama 3.1 8b, with the explanation that using a late layer would ensure that concept geometries are fully computed. They don't specify, but I assume that they did not use any of the final layers since activations in those layers tend to be more focused on the token production than the semantic content (Li and Subramani, 2025). My experiments used layer 20 (of 26) Gemma-2-2b as a roughly equivalent layer position in a model that Anthropic has released pre-trained CLT features for.
I used the same day-of-the-week input prompt format from Wurgraft et al., using "Q: What day is k days after z? \n A:" to prompt the model to produce a certain day token (and to enforce the cyclical relationships between the days). The residual stream and logits were then collected for the last token of the prompt to capture information about the predicted next token[1]. The residual stream activations,
The CLT features were computed through simple matrix multiplication as shown below, where
I followed Wurgaft et al.'s methods for fitting cubic splines through the extracted feature centroids, essentially looking to uncover a cyclical relationship between the average outputs for each of the predicted day-of-the-week representations.
Results and Discussion
As seen in the 2D PCA projection above, all three spaces produce a clearly cyclical manifold for the day-of-the-week representations, with no days that overlap or appear out of order. Out of the three projections, it is clear that the CLT features produce the cleanest circle in two dimensions, with the individual feature vectors visually more clustered around each centroid.
This cosine similarity heatmap also demonstrates that the CLT features are much more cleanly distinct and separated in CLT feature space than the representations in either the activations or behavior space[2]. The high values in activation space are not particularly surprising, since cosine similarity is frequently inflated and a poor descriptor of distributional geometry in high dimensions. CLT feature space avoids this, with each day-of-the-week centroid cleanly separated and gradually decreasing and then increasing in similarity to the other centroids as we traverse the 7-day cycle (i.e. Thursday is minimally similar to Monday, Friday is minimally similar to Tuesday, etc.). This is exactly what the CLT feature extraction method was designed to do, taking the raw residual stream and producing cleaner, more interpretable semantic features.
Normalized Arc Length between Day-of-the-Week Centroids
Mon
Tues
Wed
Thurs
Fri
Sat
Sun
Activation
0.113
0.159
0.176
0.254
0.046
0.101
0.151
CLT
0.144
0.280
0.185
0.112
0.104
0.088
0.117
Behavior
0.097
0.123
0.136
0.182
0.143
0.137
0.182
Considering the geodesic distances (the normalized arc lengths between day-of-the-week centroids shown in the table above), we actually see that the behavior space has the most regularly spaced centroids, the activation space is notably more irregular, and the CLT feature space falls somewhere in between. The irregularity in the activation space, where the weekend centroids are separated from the mid-week centroids, may be explained by research from cognitive science (Ellis et al. 2015) which shows that weekend days are processed differently than weekdays. This same research may still explain some of the irregularity in the CLT manifold, though it's less pronounced, and the weekend/weekday split is less visible in the geodesic distances.
Conclusion and Future Work
The results in this preliminary exploration largely serve to support the concepts behind the CLT feature extraction process. The same semantic relationships evident in the raw activations are present in the CLT feature space, but in a cleaner, more distinct, and more interpretable format. The fact that this cyclical manifold does exist suggests that the steering methods from Wurgraft et al. (2026) are also likely to carry over, though I've yet to complete that experiment in full.
Assuming that manifold steering is effective in CLT feature space, this opens the door to discovery and manipulation of other semantic or behavioral manifolds, using the more interpretable CLT features to identify relationships that may be better represented by a geometric manifold than a linear direction (as is currently most commonly used in steering work).
Update
Emmanuel Ameisen, from Anthropic's Interpretability team, noted that CLT features can rely on the bias term to remove a positive mean direction, so it is more reasonable to compare the CLT features with the demeaned (or mean-centered) residuals. The mean-centered cosine similarity heatmaps below demonstrate how pairwise relationships can easily be masked by a non-zero mean.
In the activation space in particular, the transformation from near-perfect cosine similarities across-the-board to only very weak similarity with neighboring days and much stronger dissimilarity (negative cosine) with maximally distant days is remarkable, though it is explained by the same distributional characteristics. With a non-zero mean, this tightly-clustered distribution is pointing in a single direction (the direction of that non-zero mean), producing very high cosine similarities. However, once the distribution is moved to the origin, all of the points are very close to zero, causing even a very small difference in one dimension to be likely to push it across the origin and bring down the overall cosine similarity.
On these new plots, the distinction between the CLT feature space and the raw activation and behavior spaces becomes much less obvious, though the CLT feature space still shows the cleanest and most evenly spaced cyclical pattern out of the three. In the CLT space, all neighboring days show positive cosine similarities and all non-neighboring days show negative cosine similarities, while in the activation and behavior spaces the (relatively) high similarities are much less cleanly associated with direct neighbors.
In the activation and behavior spaces, the weekend grouping from the Ellis et al. (2015) cognitive science citation is much more apparent in the mean-centered plots, with higher similarities across Fri-Sat-Sun. Interestingly, this doesn't hold in the CLT space, where Fri-Sat and Sat-Sun do have slightly higher cosine similarities than other neighboring days, but Fri-Sun shows the same negative cosine similarity seen in other non-neighboring days--it would seem that the CLT space's strict adjacency structure outweighs any semantic grouping within the three weekend days. It's worth noting that the behavior space has some other unexpected groupings, such as the high similarity between Mon and Wed, that deserve more detailed exploration as well.
This update complicates the original result, as it shows that some of the cleanliness of the CLT feature space was an artifact of comparing against non-zero-mean distributions. Once centered, the distinction of the CLT feature space is in producing a perfectly clean neighboring-day-similarity structure, rather than as a large gap in overall similarity patterns. This is still a real finding, but much more modest than the original framing suggested, and a good reminder to check for shared-mean artifacts when using cosine similarity in high dimensions. Thanks again to Emmanuel Ameisen for the catch.
AI Use Disclosure
Claude Code was used heavily in developing the methods in the Colab notebook associated with this post and was consulted when interpreting results and designing figures for analysis. All writing is the author's own work, and all results, analyses, and conclusions were verified by the author.
I actually ran my first experiment by using prompts like "It is [time of day] on [day of week]." and collected the activations and logits over the day-of-the-week token that was produced. Somewhat surprisingly, the results still showed a cyclical manifold (even in the behavioral space). However, the interpretation for the "representative" manifold was not as clear as that of a "predictive" manifold (especially when looking at the output logits), and there would be little to no expectation of steerability along that manifold.
See update for discussion of mean-centered cosine similarities