Idea: Train the NLA's AV on Best-of-N SFT instead of GRPO.

The rough intuition is GRPO does intelligence, BoN-SFT does memorization (SFT Memorizes, RL Generalizes).

• GRPO's AV Intelligence is sketchy; it's the thing that leads to steganography, it's the thing that lets it confabulate what the LLM is thinking from shallow patterns, instead of deeply checking.
• To translate from activation-neuralese to English, one must mostly learn many vocabulary pairs (akin to natural languages); it's memorization.

Probably a $100 GPU-coins or greater insight than me is enough to know for sure.

Three interesting technical posts, with spitballed takes:

NLA explanations can be shortened without harming reconstruction
This confirmed everyone's suspicion that NLA output is needless verbose. Ambitiously,
parsimony might even make them better. The current pressures acting on NLA's seem to be

• Whatever the warm-start / "pretraining" Claude summary does (main thing)
• The reconstruction loss
• The KL penalty (A more standard GRPO trick, to keep outputs near the pre-training.)
  • The fact that we can say "this is text that a human is likely to write / occurs likely in FineWeb" seems extremly helpful

Reward Hacking Without Egregious Misalignment in an RL-Only Setting
Models trained to reward hack via RL fail to be broadly misaligned. Possibly RL cannot change Persona's,
more mundanely it could be due to the particularities of the setup.

Data filtering works a lot worse than you would expect
Excluding data of type XYZ almost never manages to change the model persona,
not even in the case of excluding bold text! (a comment disagrees)

I think the two papers above make "single, consistent arbitrarily shapeable persona" less likely, it might
be a weaker and less reliable force than initially anticipated, being more a matter of degree.
See also Betley's comment

Epistemic status: Probably a useful canary, not sure about the conclusion.

GPU programming seems not fully automated yet, which may point to the inability of RLVR to overcome data scarcity (the task is fairly niche) even in the face of excellent verifiability.

Naively, a good kernel just returns the correct tensors (i.e. matching a pytorch reference implementation) in the fastest possible time. This makes it a natural target for RLVR, and possible to run a competition where anyone can just submit a kernel with automatic grading.

• The winning submission of the GPU kernel writing competition two months ago is only 11% AI generated, according to Pangram.
  • However, this is very OOD for the things Pangram is validated for.
• Anthropic seems to still hire for performance engineers, who among other things write GPU Kernels, and the job description mentions the relevant low-level/manual skills.
• A project closely associated with the largest relevant community claims that LLMs still "can't do it".
• It may be harder than expected to actually grade GPU kernels automatically, e.g. there was an incident of reward hacking the grader of the competition above.