Where can I find a post or article arguing that the internal cognitive model of contemporary LLMs is quite alien, strange, non-human, even though they are trained on human text and produce human-like answers, which are rendered "friendly" by RLHF?
To be clear, I am not asking about the following, which I am familiar with:
Eliezer Yudkowsky's description of evolution as Azathoth, the blind idiot god, as a way of showing that "intelligences" can be quite incomprehensible
The difference in environments between the training and the runtime phase of an LLM
The fact that machine-learning systems like LLMs are not really neuromorphic; they are structured differently from human brains (though that fact does not exclude the possibility of similarly on a logical level)
Rather, I am looking for a discussion of evidence that the LLMs internal "true" motivation or reasoning system is very different from human, despite the human output, and that in outlying environmental conditions, very different from the training environment, it will behave very differently. A good argument might analyze bits of weird inhuman behavior to try to infer the internal model.
(All I found on the shoggoth idea on LessWrong is this article contrasts the idea of the shoggoth with the idea that there is no coherent model, but does not explain why we might think that there is an alien cognitive model. This one likewise mentions the idea but does not argue for its correctness.)
[Edit: Another user corrected my spelling: shoggoth, not shuggoth.]
First, a factual statement that is true to the best of my knowledge: LLM state, that is used to produce probability distribution for the next token, is completely determined by the state of its input buffer (plus a bit of indeterminism due to parallel processing and non-associativity of floating point arithmetic).
That is LLM can pass only a single token (around 2 bytes) to its future self. That follows from the above.
What comes next is a plausible (to me) speculation.
For humans what's passed to our future self is most likely much more that a single token. That is a state of the human brain that leads to writing (or uttering) the next word most likely cannot be derived from a small subset of a previous state plus a last written word (that is state of the brain changes not only because we had written or said a word, but by other means too).
This difference can lead to completely different processes that LLM uses to mimic human output, that is potential shoggethification. But to be the real shoggoth LLM also needs a way to covertly update its shoggoth state, that is the part of its state that can lead to inhuman behavior. Output buffer is the only thing it has to maintain state, so the shoggoth state should be steganographically encoded in it, thus severely limiting its information density and update rate.
I wonder how a shoggoth state may arise at all, but it might be my lack of imagination.
Where can I find a post or article arguing that the internal cognitive model of contemporary LLMs is quite alien, strange, non-human, even though they are trained on human text and produce human-like answers, which are rendered "friendly" by RLHF?
To be clear, I am not asking about the following, which I am familiar with:
Rather, I am looking for a discussion of evidence that the LLMs internal "true" motivation or reasoning system is very different from human, despite the human output, and that in outlying environmental conditions, very different from the training environment, it will behave very differently. A good argument might analyze bits of weird inhuman behavior to try to infer the internal model.
(All I found on the shoggoth idea on LessWrong is this article contrasts the idea of the shoggoth with the idea that there is no coherent model, but does not explain why we might think that there is an alien cognitive model. This one likewise mentions the idea but does not argue for its correctness.)
[Edit: Another user corrected my spelling: shoggoth, not shuggoth.]