This is a special post for quick takes by J Bostock. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Pro tip for all fine-tuning experiments. Fuck around with your optimizer. In some cases I've seen different results with Adam and Muon. If you want to use LoRA there's a Muon version called Riamannion or something like that. From a physics-of-LLMs perspective, Muon is a very different creature to Adam; I'd like to get as much tacit knowledge into the community as possible, about exactly what this means.
If a result doesn't generalize between optimizers, that often is a reason I'm much less interested in it, because this is evidence against it generalizing to future situations I care about. This is especially true for research that tries to learn facts, rather than research that tries to produce models with particular properties so that they can be used as subjects of research. E.g. it's very hard to make sandbagging model organisms, and it would be sort of handy to have those model organisms around, so it is useful to know whether different optimizers hel... (read more)
Fig 1 of the paper show the opposite, right? SGD causing less EM than Muon.
12J Bostock
I have a horrible recurring thought that in the small number of worlds where we we actually figure out some physics-of-LLMs persona-based alignment strategy which does some kinda leveraging of the pretrained prior into an alignment solution, there's an even smaller set of worlds where we die anyway because a research engineer flips a config from --adamw to --muon to squeeze an extra 2% performance out of the GPUs.