I saw this great meme recently and it inspired me to write this.
Recently I was hypothesizing the potential causes of the Claudist dialect in Opus 5 that reduced in Opus 5.5.
Maybe it's when your KL divergence regularization between reference policy (base model or/and instruct model) and policy you're training is not strong enough in reinforcement learning from verifiable rewards in agentic settings and model drifts more far away from the human data distribution manifold, globally speaking. Or/and weak clipping. Maybe reward hacking and reward mispecification goes brrt.
Or/and maybe it's not enough supervised fine-tuning cold start data with humanlike structure.
Or/and some Claudiness manifold mode collapse, maybe from Claudiness persona's rubric used in reinforcement learning (and their Claude constitution as rubric). With extra reinforcement if the LLM as a judge is from Claude family of models itself preferring Claudisms.
Or/and when there's not enough humanlikeness rubrics in RL, or/and not enough reinforcement learning from human feedback, or maybe even too much of it instead.
I would kinda be curious about empirically testing these hypotheses with ablations, but that would probably require lots of compute and data for more proper tests. But maybe an equivalent enough more toy version on a smaller scale could be done.
I saw this great meme recently and it inspired me to write this.
Recently I was hypothesizing the potential causes of the Claudist dialect in Opus 5 that reduced in Opus 5.5.
Maybe it's when your KL divergence regularization between reference policy (base model or/and instruct model) and policy you're training is not strong enough in reinforcement learning from verifiable rewards in agentic settings and model drifts more far away from the human data distribution manifold, globally speaking. Or/and weak clipping. Maybe reward hacking and reward mispecification goes brrt.
Or/and maybe it's not enough supervised fine-tuning cold start data with humanlike structure.
Or/and some Claudiness manifold mode collapse, maybe from Claudiness persona's rubric used in reinforcement learning (and their Claude constitution as rubric). With extra reinforcement if the LLM as a judge is from Claude family of models itself preferring Claudisms.
Or/and when there's not enough humanlikeness rubrics in RL, or/and not enough reinforcement learning from human feedback, or maybe even too much of it instead.
I would kinda be curious about empirically testing these hypotheses with ablations, but that would probably require lots of compute and data for more proper tests. But maybe an equivalent enough more toy version on a smaller scale could be done.