Really enjoyed this paper/post. Building upon @simon's entrenchment point and also @Charles Ye's observation about how LLM output pollutes crawls - I think there is a frame that ties them together and it makes the style-over-tag result look inevitable, not just like a failure.
The result is what loss predicts. A role tag is really a rare and low-variance token. While style is a high-frequency signal and it's independently predictive of the same downstream tokens that the tag is supposed to gate. SGD keys on features that carry more predictive signal per uni... (read more)
Really enjoyed this paper/post. Building upon @simon's entrenchment point and also @Charles Ye's observation about how LLM output pollutes crawls - I think there is a frame that ties them together and it makes the style-over-tag result look inevitable, not just like a failure.
The result is what loss predicts. A role tag is really a rare and low-variance token. While style is a high-frequency signal and it's independently predictive of the same downstream tokens that the tag is supposed to gate. SGD keys on features that carry more predictive signal per uni... (read more)