Thanks for the reply! Two points.
In your original comment you said “LLMs can’t form and use novel concepts”. However, in your explanation you seem to be saying something different. I would summarise what you say there as: "Sure, LLMs can form and use new concepts. However, the degree to which they can generate new ideas and then continue to develop new ideas on top of those (serial complexity) is limited by their context window because they can't do continual learning." Is that a fair summary of what you are saying?
The second point is that even if you agree that continual learning is an important bottleneck, the question remains how far you can get without it. You say that a current LLM "rapidly hits limits if you want to build giant new edifices of new concepts built on top of new concepts etc." My question is, how do you view things like OpenAI solving Navier-Stokes by having 10,000 agents cooperate, work on different tasks, communicate etc.? It seems to me like they managed to produce a result with a pretty high degree of serial complexity by doing so. (Like you say, this doesn't mean that they then were able to integrate that into their weights a la continual learning, but the output certainly seems to have had a high degree of serial complexity)
My underlying argument is that "form and use novel concepts" or 'serial complexity' might not be the right metrics to think about. Also LLMs already seem to be able to do both. I think that the thing people care about when they ask if LLMs can be creative is something different, more what you were gesturing at with your linear algebra experiment. My hypothesis is that often when people say things like LLMs are 'not creative' or 'can't form novel concepts' or 'can't achieve high serial complexity' they have something else in mind, but putting that thing into words is actually difficult. One framework that seems to come closer to putting into words what people mean is Margaret Boden's three levels of creativity.
Is this what you were getting at with your linear algebra experiment?
Many people expect the AI paradigm to approximately AGI complete, and expect it to smoothly scale to AI that can do innovative science. i.e. the kind of science you'd need to do to develop a plague from scratch that actually kills all humans, with very limited opportunity to experiment. Or, the kind of science you'd need to solve alignment, so you could build a more powerful successor AI that shares your values.
A periodic thread of disagreement has been @TsviBT, @Steven Byrnes and some others arguing that, yes, the AIs are getting better at verifiable tasks, but they still seem to be missing some basic sauce that lets them actually form new concepts on the fly and apply them.
The AIs are getting better at "brute force creativity", where you try every single idea ever devised by Man in parallel and maybe one of them works. But there's still a kind of creativity that a) would have way higher efficiency and b) is maybe necessary for doing some kinds of breakthrough work.
On one hand, my subjective experience has been seeing AIs be able to demonstrate increasingly interesting judgement and taste in increasingly many domains (even open-ended ones). But, I do indeed still run into AIs that get confused and "just don't get it", in a way that is pretty suggestive that there is still something significant missing.
OpenAI has lately been shipping some major results in mathematics, for problems that people had previously agreed were important and hard. But, I'd heard for previous results that the proofs turned out sort of "not actually interesting", compared to how sometimes makes you go "Holy shit that's beautiful/elegant/surprising" and conveys someone is a level above you."
Recently they shipped a lot of new proofs on important problems.
I'm not a math guy. But, seems fairly important for some people with enough context to wade through them and figure out "What sort of cognition was involved here?". Do any of the proofs have the vibe of AlphaGo's Move 37? Does it invent a new concept that does real work?
I'm guessing answering this is a nontrivial task that'll involve some... archaeology? Might be cool for a competent math theorist who doesn't have a higher priority project to organize the labor. (I'm guessing the broader math community will do this on their own, but not necessarily organize the information in a way that's optimized for figuring out "is this the kind of mind that could solve its own alignment or quickly develop radically more powerful weapons than we have available?").
I'm shamelessly copying in @DaemonicSigil's quick take about this, to give some hooks for people to start thinking through.