I would go further and say that this is a lot of bad and not a lot of good, unless this directly leads to some version of "we can explain the model well enough to hand-write a computer program that is functionally nearly identical." I have to imagine that this type of approach will let us reach a strong form of RSI far sooner than it will let us reach that interpretability goal. While the ability to use this to reduce some features associated with misalignment is nice, if we're ultimately still in the regime of "we can tweak some things to make the model better without actually knowing how the whole thing works," then we still face the issue of suppressing the most visible forms of misalignment while letting the harder to detect ones develop.

Writing weights from symbolic programs (re experiment 5) is also not an entirely novel capability; cf Tracr It is of course interesting that llms can do this without having defined the spec in some formal language, but this doesn't seem any more unprecedented than recent math results. That llms are soon to be better optimizers than Adam/SGD is speculative