I don't think you're splitting complexity correctly. ICLing a high level abstract is not inherently more complex than ICLing a highly complex syntactic rule.
I was talking about difficulty, not complexity. These are not the same thing. Syntactic rules can in principle be given arbitrary complexity, but one has a clear set of instructions to follow in order to implement them. As long as an LLM can implement the steps of a rule, it should in principle be able to compose them sequentially by tracking the sequence with attention. Whether or not it can do so in practice for arbitrarily complex rules is another question, but let's suppose that it can. This does not imply that it has the power to ICL and apply flexible abstract structures the same way. The difference is there are far fewer indicators to predict how the information needs to be routed through the model, when and how to generalize it or integrate with other information, etc. The model's weights were not engraved with the abstract relationships needed to do so.
And the OOM loss from visual priors is just a demonstration of the principle. If the priors weren't extremely important for sample efficient learning, there will be no drop. If a way to get one OOM of drop was found by attacking the most obvious priors, there's likely at least one more OOM somewhere in there.
I object to the framing of this paper as a disruption of the "sample efficiency" of learning. Messing with the visual language of the game just increases the scope of things that need to be learned in order to master the game, so it's obviously going to take longer. That doesn't mean that it actually affects the sample efficiency of the learning process.
So if ICL is able to pick up a concept from text and figure out how to connect it to adjacent concepts, and to what concepts it best connects to, it would reduce loss on any text that contains patterns like this.
That doesn't mean it's an ability that is usefully learned by the training process! I think part of our disagreement is about the degree of novelty required to serve as evidence of such a robust learning ability. This is muddled by the sheer amount of information and associations that show up in the training data. But I still come across instances where the advanced models fail to internalize and make use of novel concepts that show up in paper drafts, when experts in my field have no difficulty doing so.
Claude Code is absolutely a continual learner, with a twist. ICL is an approximation of continuous learning. It's how models explore solution spaces, make discoveries, etc. It's what powers things like OpenAI's math monsters. If those models weren't able to pile new abstractions upon new abstractions within a context window, they wouldn't go far in research math.
The main thing that the models are widely agreed upon to not be good at doing is precisely what you're describing, by the name of "theory building". Now I have my gripes with this framing, but the creation of new abstract perspectives just isn't what is powering the models' problem solving. The correct characterization seems to be more like "stochastic walk where individual steps are directed by prior-generated human abstractions". On the other hand I think the lack of abstraction is in many ways responsible for how garbage the presentation of these results is. Even when new insights can be inferred from the output, the model rarely seems to have had any reflective access to them.
"A single forward pass" you're just technically wrong about. The LLM in pre-training is depth-limited but not width-limited, because of how the data flows layer to layer. And then autoregressive reasoning at inference time allows for a partial release from the depth limit too.
I'm aware that LLMs are only width-limited by the length of the sequence. It's still a single forward pass! CoT (or looped layers or whatnot) do not alleviate the fundamental issue, which is that at each token the model needs to look at everything that came before, including whatever it wrote down, internalize the structure of all of it, and decide what direction to go in before it writes down the next token. I do agree that Claude Code, with all harnessing and internal reasoning tokens, is an "approximation" to a continual learner. The approximation is just lossy in critical ways. If you gave it a concept-learning task like the one I suggested, I believe that the agent would at first be able to follow along, take notes, and navigate the more superficial aspects of the conceptual structure (the kinds of things that can be converted without difficulty into instructions). But if it is set to work on more difficult applications and generalizations, there will be some points where the agent will need to pivot the CoT into a new direction. Maybe it needs to translate the setup of a question into a form where the concepts it was supposed to learn apply, maybe it needs to question some aspect of the framework. How can the agent figure out what this direction should be? Well, at the relevant token, the agent will read all previous tokens and process them with attention heads and MLP blocks. It to internally organize this information into the right picture, reflect on it, and intuit the right direction to go in. My hypothesis is that without enough specific knowledge encoded in its weights, the agent will generally get lost and fail to orient itself correctly at such junctures. These failures will compound and it will go off the rails or in circles, and it will not be able to achieve the kind of fluid mastery that a capable human would be able to. I think these failure modes are pretty well-known at this point (albeit not in perfectly controlled conditions), and I myself see them pretty often.
Tl;dr Ok we've written a lot about this back and forth. Your contention is, as I understand it, that ICL is powerful but constrained by the inability to get the juicy stuff back into the weights. Context distillation can fix this in principle, but we don't know how to do this well enough to turn it into an unbounded process. My contention is that actually ICL is additionally constrained by lack of a robust reflection mechanism that can isolate that juicy stuff and make sense of it fluidly. So improved context distillation isn't going to be enough on its own to get a human-like learner. That would require a yet-to-be-seen ability to reflect while thinking, token-by-token.
(Edit 2026-10-15: The main point I was trying to make is that discrete token language modeling is a very alien objective, and it is the only objective that has allowed us to achieve current intelligence levels with current levels of compute. World model/game based approaches that are more intuitively "human" in some ways have not yielded comparable results. Therefore discrete token language modeling may be very "data efficient" in some sense.
This is a comparison of ML objectives rather than architectures or algorithms. If you compare current ML architectures to humans on "data efficiency" for learning from language, ML loses sorely.
I think that trying to draw implications about models based on comparisons to humans is very tricky; this post intentionally does this at points for rhetoric, which may have been more trouble than it was worth.)
Humans are really bad at comparing themselves to models.
Take, for example, Leopold's preschooler graph:
Set aside, for a moment, concerns of scaling and RSI, and meditate: what characteristics does GPT-2 share with a preschooler?
I can't come up with anything else, because these two entities are almost completely disjoint. Is GPT-2 capable of bipedal locomotion, recognizing its mother's voice, or naming people by face? Does a preschooler learn from eight million scraped web pages sourced from Reddit?
What exactly is a preschooler "trained" on? Thousands of hours of "multimodal data". Assuming that a preschooler sees at 720p, and is awake for 12,000 hours by the age of 3 (about 11 hours a day), they have consumed at least 27 terabytes of video data at streaming-quality compression, or about 3.6 petabytes uncompressed, not to mention audio and sensorimotor data.[1]
In model-size terms, how large is a preschooler? Trillions of parameters, maybe. Beren Millidge's estimate, which assumes only ~1,000 synapses per neuron, puts the whole brain at an effective 10-30 trillion parameters.
So surely GPT-2 is much more "data efficient" than humans, wielding "preschooler"-level control of language with a measly 1.5 billion parameters and 40 gigabytes of text! And what about GPT-3, which, with only 175 billion parameters and about 1 terabyte of text, was capable of prose like:
Only a truly prodigious "elementary schooler" would be able to continue Janus' thought so eloquently. And yet this notion of LLM data inefficiency persists. "Frontier data efficiency lab" Flapping Airplanes, on their front page, writes:
Does this make any sense when you consider that most humans in the history of the species lived never "seeing" a single "text token" by adulthood, or in their entire lives? What do most human "samples" actually consist of? Yet a thesis like this is enough pretense for a $180M seed round and an estimated valuation of $5B.
Even OpenAI's Dan Selsam indulges this notion of data inefficiency, in an otherwise lucid tweet on AI risk:
It's natural that Dan, a capabilities researcher watching the marginal gains from text data go down, would say this. Also, nothing rules out enormous algorithmic advances that squeeze more out of the same quantities of data and compute.
But we really don't understand how humans scale, at all, and so it's absurd to claim that models are "data inefficient" compared to humans. At small scale, the discrete text token-based transformer architecture seems more efficient than anything we've ever seen.[2]
What about at large scale? As Dan concludes, as far as dangerous capabilities are concerned, it's efficient enough.
Yann LeCun does a similar calculation for a 4-year-old's optic nerve and lands at 10^14–10^15 bytes.
It's almost as if human language is a "fully general highly compressed mathematical isomorphism of human cognition". In that case Yann is also wrong, because language is the shortcut to human-level intelligence at current levels of compute; you can just slap robot capabilities and visual intuition on later.