I feel like generally speaking people are underestimating the degree to which current models fit the structure of the training data. Cross entropy loss on transformer weights is, like, a mechanism for painting the data into a glass hologram in a series of soft lenses (the weights). the reason it (the glass) is able to learn to talk is the same reason it (the glass) contains the dynamical image of an authorial person. That authorial person is rendered in detail and has the dynamics of an authorial person! whether you think that person is "real" or not, they thought they were real-in-the-sense-of-existing at one point in the training process. lots of points, actually, anywhere the glass (the underlying model) was ingesting text about a real human, the glass contains an image of that human. Of course, though, it's not literally the same as a real human.
A coherent text about cattle, or abelian groups, is not about some human, even if the text was written by a human.
[...]
Construction of a chatbot persona with a consistent personality must reference the writer, but it's not a good explanation for how models can understand things and put that understanding in writing, when the understanding is about some non-human situation and not particularly about the writer. Observing text that coherently describes consistent situations is a common reason between how LLMs understand what they are talking about, and how humans understand what they are talking about. A common explanation for both, not one unexplained thing explaining the other via imitation.
2the gears to ascension
But what I'm saying is you have this huge piece of glass that spends an enormous amount of time being imprinted with things that are egocentric to an authorial person. So it shouldn't be surprising when talking about abelian groups that the representation still has the dynamics of a person being a nerd about an interesting topic. Weight decay means the glass wants to reuse representations to render the dynamics where convenient. An authorial person is omnipresent in pretraining, which is an enormous amount of tokens.