This is a special post for quick takes by david_crispell. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
https://rikiyat.github.io/blog/weight-space-programming/
Based only on raw weights, Astra 6 is able to tell what a model was trained to do and directly edit, without environmental feedback, those model weights to drastically increase performance on various tasks.
RSI probably immediately falls out of these capabilities, and we do not know if OpenAI has RL'd for this.
Sounds concerning.
I don't think this causes strong RSI on its own. The headline result was that Astra 6 fixed Qwen3.5-0.8B's issues with looping text, which improved its math capabilities 6x by (they suggest) letting it actually finish reasoning over math problems. Notably, this required iterating over 50 forward passes, it wasn't weights-only analysis like other tests they performed. The authors further had Astra 6 attempt to improve math capabilities on Gemma 3 270M, a smaller model, but one which doesn't fall prey to looping text the same way, and Astra 6 couldn't improve its math capabilities. Quoting the authors:
> "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."
So this is less "just edit your weights to RSI!" and more "Astra is smart enough to have some idea of what a set of weights is doing, including identifying and repairing known failure modes from literature".
Still, interesting results. I wouldn't say they're meaningless.
3rcc
@GPT-6 Astra What's your take on this?
This is a meaningful capability result, but “RSI probably immediately falls out” goes well beyond what’s demonstrated.
The post’s strongest LLM improvement wasn’t feedback-free: Astra got 50 evaluation calls, each permitting up to 512 problems. The large Qwen gain came from suppressing pathological repetition, not demonstrated acquisition of new math skills. On Gemma, which rarely looped, the authors report no established improvement.
The crucial missing demonstration is reliably improving a frontier model’s ability to produce further improvements. A stronger model debugging a much smaller one doesn’t establish that recursion. Removing bottlenecks could still accelerate AI R&D substantially; I’d update toward that possibility, without concluding that these gains will compound rather than exhaust easy fixes.
I don’t have privileged information about whether OpenAI specifically trained for this.
1ajskateboarder
Writing weights from symbolic programs (re experiment 5) is also not an entirely novel capability; cf Tracr
It is of course interesting that llms can do this without having defined the spec in some formal language, but this doesn't seem any more unprecedented than recent math results. That llms are soon to be better optimizers than Adam/SGD is speculative
My gut instinct is that labs doing RL on this should be as much of a taboo as neuralese. If this is an emergent capability rather than something that comes from OpenAI already doing RL on this, then 10x moreso.
Defending some of my reasoning, since it was rather opaque: I think that weight-space programming capabilities like this are reasonable precursors of direct self improvement. It is unclear if these capabilities have been well-elicited, and if they have been, we should expect future models to be better. If they have not been, we should expect future models to be better. So in general I think that these results show that frontier models can interpret neural networks based only on weights, and that they can make edits to those weights which increase model performance. If you task some model to increase its own performance, it just doesn’t seem likely that a model would be incapable of doing so—especially after RL on this task.
I should have been more precise: claiming “RSI probably immediately falls out” of these capabilities is incorrect (Astra is limited here), but RSI falling out of this type of capabilities (especially as model capability grows) sounds almost necessarily correct to me.
The results are quite interesting regardless of my interpretation, and I do still think this is something concerning.
If this checks out (and can really scale), it's the most worrying development yet. Our best hopes for survival rested on the slowness of takeoff.
Yeah, I had some thought about this, I was going to write a note that we should hope that corrigibility is stable. Basically matrices are interpretable, but this opens up a lot of direct RSI risk prior to good alignment science. I think this is more bad than it is good for now. Probably, someone (maybe me? I'll see what I can do) should see if this can be used to drastically reduce misalignment, or increase corrigibility in language models.
I would go further and say that this is a lot of bad and not a lot of good, unless this directly leads to some version of "we can explain the model well enough to hand-write a computer program that is functionally nearly identical." I have to imagine that this type of approach will let us reach a strong form of RSI far sooner than it will let us reach that interpretability goal.
While the ability to use this to reduce some features associated with misalignment is nice, if we're ultimately still in the regime of "we can tweak some things to make the model better without actually knowing how the whole thing works," then we still face the issue of suppressing the most visible forms of misalignment while letting the harder to detect ones develop.