I don't think this causes strong RSI on its own. The headline result was that Astra 6 fixed Qwen3.5-0.8B's issues with looping text, which improved its math capabilities 6x by (they suggest) letting it actually finish reasoning over math problems. Notably, this required iterating over 50 forward passes, it wasn't weights-only analysis like other tests they performed. The authors further had Astra 6 attempt to improve math capabilities on Gemma 3 270M, a smaller model, but one which doesn't fall prey to looping text the same way, and Astra 6 couldn't improve its math capabilities. Quoting the authors:

> "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."

So this is less "just edit your weights to RSI!" and more "Astra is smart enough to have some idea of what a set of weights is doing, including identifying and repairing known failure modes from literature".

Still, interesting results. I wouldn't say they're meaningless.

This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted!

This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!

-- Neel Nanda, seems to think this isn't too surprising https://x.com/NeelNanda5/status/2117375052618539485 .

Defending some of my reasoning, since it was rather opaque: I think that weight-space programming capabilities like this are reasonable precursors of direct self improvement. It is unclear if these capabilities have been well-elicited, but if they have been, we should expect future models to be better. If they have not been, we should expect future models to be better. So in general I think that these results show that frontier models can interpret neural networks based only on weights, and that they can make edits to those weights which increase model performance. If you task some model to increase its own performance, it just doesn’t seem likely that a model would be incapable of doing so—especially after RL on this task.

I should have been more precise: claiming “RSI probably immediately falls out” of these capabilities is incorrect (Astra is limited here), but RSI falling out of this type of capabilities (especially as model capability grows) sounds almost necessarily correct to me.

The results are quite interesting regardless of my interpretation, and I do still think this is something concerning.

My gut instinct is that labs doing RL on this should be as much of a taboo as neuralese. If this is an emergent capability rather than something that comes from OpenAI already doing RL on this, then 10x moreso.

drastically increase performance on various tasks

This was basically just a looping fix. They say that about 93% of the answers hit the token limit before this edit. And after Astra switched that off, it improved to 33%. When the same thing was tried on Gemma, which doesn't loop, nothing improved. In their words, editing "could not add an ability the model lacked"

If these capabilities can be improved through RL, then I think we have reason for concern

I share your concern, but for a different reason. RSI means adding new capabilities, and so far this only shows it can remove things that get in the way. If more training makes this better, I'd expect the removing side to improve first. That worries me for its own reason: safety training could look like one more thing to switch off

It seems to imply that at least those larger models have a representation of weight subspaces. I think it is not generally surprising to see some form of representation for inner activation patterns. The J space is also an example of this. An explicit way of forcing something like that was discussed in Self-prediction acts as an emergent regularizer. But representation of representation doesn't explain why it can interpret the weight values. Generalizing introspection patterns doesn't give you access to your weights. I see the following explanations:

• The models obviously have been trained on a lot of LLM papers and code etc and represent that knowledge deep enough to grok the relation between arbitrary concepts and weights to represent it. That would imply an inner optimizer, which I find worrisome.
• Dito but without generalizing from LLM papers and code specfically but pure generalization (groking weights even deeper). Even more worrisome if true, but think this is unlikely.
• This specific model may have been trained on exactly these kinds of correspondences by the lab! It seems at least plausible that the labs may have tries such a thing. It seems useful for interpretability work. Maybe it was not intended to be used this way by external parties, but whatever. This would be less worrisome as it seems more like a "normal" trained capability. It could still lead to an inner optimizer, but via a less spontaneous route.