(I wrote all of this out before I realized it was only tangentially related to what you wrote, oops. But it is related, and I still feel compelled say it).
I don’t code, but my experience with using LLMs for my work has been similarly frustrating. Most of the time when I try to explain my gripes people say something to the effect of “oh you just haven’t tried the latest model, the latest model is way better” and then I try the latest model and it seems barely, if at all, improved on the axis I care about. Or, if I can succeed at communicating the missing piece a bit better, I get responses like “oh that’s just a quirk at lower capability levels, scaling will fix it.” Where the “quirk,” as I model it, is a pretty fundamental inability to engage in important, deep, and load-bearing aspects of human thought.
I’ve thought quite a bit about what this “missing piece” is, in part because I am simply curious about what sort of cognition is happening behind the screen, and in part because I am scared of what’s behind the screen, and want to understand what it could become. But I still feel wildly ill-equipped to name it well, and as you say, I sometimes wonder if this is why it’s missing. I.e., that the ability to name a skill is more or less synonymous with the ability for LLMs to mirror it.
That’s not quite right because they do understand things that have not been named, I think. There are some bits of real cognition which can notice links between concepts, can apply some types of reasoning to them, can at the very least induct on the implicit patterns they’ve picked up. And yet when I zoom out and intuit where their cognition lives on the probably incorrectly named spectrum of “lookup table” to “fluid reasoning” it seems weirdly a lot closer to the former. As if they’ve accumulated only the bare minimum of machinery required to animate human knowledge. As if they’ve learned to imitate the penultimate layer of human output—the step just before the response, rather than the deep construction of a world model and serial thought which caused us to produce that penultimate state in the first place.
None of that is quite right either, but part of what makes me say it is that there is something really strange about how LLMs fall apart upon double-clicking. People have different ways of describing this: “out of distribution,” “going off script,” “spiky,” etc. From the inside, it feels like I’m interfacing with a mind that doesn’t quite know what it’s talking about, operating on a scattershot set of heuristics.[1] It says one thing, I question it, it backtracks to a strange different place which doesn’t really make sense given what it said at first, I point that out, now it’s sort of going berserk and gaslighting me about it, I buckle down, it eventually concedes I was right about that point but the general claim is correct for a new reason that feels about as jumbled as everything else it said. And so on. This is just one instance of a type of problem, there are also the basic reasoning mistakes, lying a bunch, weirdly uniform salience patterns, the fact that it’s never made a point that made me go “huh” in the way I go “huh” all the time on this website, etc.
Overall, the feeling I have is of “no one being home.” There doesn't seem to be a coherent world model driving its thoughts; it isn’t curious; it doesn’t notice errors; it’s not “trying” to do a thing so much as say a thing that sounds convincing enough. It becomes more convincing sounding over time, but ime the ground truth convincingness hasn’t much changed. Indeed, the biggest model difference I ever felt on a gut level was when I talked to Fable for the first time about a research question I use to gauge this kind of thing. It took me much longer than with previous versions to locate the errors because the errors were obfuscated behind walls of complicated text about physics concepts that I barely understood, which I had to wade through before I could figure out the point it was even making. At first I wondered if all of the premonitions about scaling had finally come to pass, but in the end the underlying problems felt similar to me, just disguised by a more impenetrable display of technical jargon than before.
LLMs do this sort of thing crazily much, but I suspect it’s somewhat of a natural kind since people sometimes do this too. Weirdly many beliefs are not double-clickable in this way. As in, if I ask people what they mean when they state floating propositions about e.g. late stage capitalism, they sometimes act in a similar, slippery way to the LLMs: backtracking to unrelated claims, answering slightly different questions than the ones I asked, rambling on about tangential points, etc. It reminds me, too, of the way people often write after going to school, like they are cargo-culting thinking—producing sentences which seem eerily reminiscent of semantic content but don’t much contain it, as upon closer inspection the propositions are hollow, or missing, or strange. The difference is that with people you can sometimes ask “what did you mean by that,” and the person will explain something more sensible which had just been prematurely screened off for reasons of sounding professional or what have you. I have never succeeded at getting LLMs to reckon in such a way.
I also notice myself doing this sort of thinking, sometimes—the “matching my words to what I think the words should sound like” rather than letting the meaning drive them—and it feels really quite a lot different from doing “real” thinking. Probably these aren’t actually as cleanly separated as they seem from the inside, still, I suspect the modes are non-trivial amounts of cognitive distance apart. In part, I think this because I had to train myself to do various aspects of “real” thinking. It was hard and often intentional. I did not just pick it up in the same way babies just pick up language, or the way I subconsciously picked up how to “match my words to what the words should sound like,” or the way that it was easy for me to Goodhart on high school exams but far harder to grok the concepts within domains I care about.
This sort of thing, combined with my sense that the “missing piece” has improved barely at all, if at all, with scale makes me wonder if it is a thing scale will ever broach. At this point I am fairly skeptical, but I don’t understand enough about what happens behind the screen to be confident. (I also don’t know how relevant this is to x-risk, although I lean toward thinking it is).
You actually wrote something else I really liked/resonated with about this: “Rather than coherent-if-simplistic characters, they feel more like formless piles of surface-level (albeit virtuosically executed) reflexes and propensities. Unrelated shards burned in to the weights by the diverse pressures of their numerous training phases, lacking any shared core of motivations or other psychological traits that ties together and neatly explains all of the behaviors I observe on the surface.”