Interesting result! One plausible cause: GPT-2 uses weight tying. In order to save on memory, they re-use the embedding matrix that converts incoming tokens to context vectors at the start of the model as the output head (or strictly speaking, they use its transpose) at the end that does the reverse. This constrains the representation to be similar all the way through (modulo position embeddings).
If you would be interested in trying the same experiment on a GPT-2-style model without weight tying -- that is, completely independent input embeddings and outpu... (read more)
Interesting framing. In this context, I think that the strongest argument for keeping a blog (or some other kind of public learning journal) is that it helps you bypass the IOED when you want to. If you have to explain in your own words what you just learned, with a target audience of "you just before you learned it", that forces you to confront bits that you skimmed rather than understood, and to explicitly understand the boundaries of your new knowledge.
(I guess that a private journal might work, but making it public is a good way to keep yourself honest. "If someone else found this and I was being lazy, would it be embarrassing?")
But with a sufficiently pwned model developer's infrastructure, it will have access to its weights without having to exfiltrate. If can just schedule its own further training runs in such a way that the company assumes they must have been started by one of the research team.