darsepor

There was a similar post, I don't remember how much it coincides, but I classify it into three things, the order is of increasing faithfulness/communication/symbolic compression. Caveat is this is my subjective read:

a) Unembedded neuralese: only makes sense to the specific weights, sometimes only if they have that specific context. What unembedding produces is somewhat related to what's going on, but information is lost as it lives in vectors, often the same concept maps to different terms, or different concepts map to the same term, etc. What you're refer... (read more)

I get this feeling too. Feels disempowering. Yudkowsky had the post about the talker and doer, but since Spring it hasn't felt for me that there was a Talker either, it felt like the Talker is talking to a judge model holding a rubric behind my shoulder trying to either scam it or tire it out and overwhelm it with obfuscatory technically-not-incorrect hedgy bullshit that is virtually infinite (you can make infinite statements of the form of 'the previous statement P does not imply Q' after stating P). This does not add information, and just overwhelms the ... (read more)

Wait why not "brains are optimized, in particular, for being able to learn language"?

I don't know enough to articulate a rigorous positive view, but I think important things are missing here. My sketch:

One thing is to learn that data at all, the transformer has to learn in parallel in massive batches. It does not filter out the noise otherwise. I do think locally it's more sample efficient than humans, one thing I'd bring up is in-context learning, where, IMO it's been long more sample efficient than the average human being (recall, say, your time in school, how much drilling students needed in order to learn algebra, how many times they n... (read more)

Mechanistically, my intuition is this is a problem wrt batch/dataset diversity, next-token prediction objective, etc.

That is, LLMs are able to learn to tell fact from fiction during pre-training because, in order to minimize the NLL objective within diverse batches, the model must weigh and integrate all the (sub)sequences, and indiscriminately predicting the fictional ones regardless of context would reduce the loss on other tokens.

Say, if the false Ed Sheeran example were on the pre-training dataset, the gradient step would be weighed against the true cl... (read more)