It seems that we disagree about how powerful ICL actually is.
I agree with what you're saying about ICL+distillation not being that different from RL in spirit. I just also believe that both of these are pretty limited in terms of what they can actually teach an LLM to do without appropriate priors encoded in the base model. This doesn't apply to humans! It seems really important that as we generate a trajectory this backreacts on our priors in a way that reshapes the future of that trajectory. This means that the trajectories we generate are way less limited by our priors. Rather than needing a bunch of external signal to select between parallel rollouts that comes from a verifier or a human selecting context, we can bootstrap ourselves into learning complicated concepts (including ones that nobody has figured out before).
I also agree with your model that we come prewired with exactly the right information that provides us with a learning algorithm that then stacks knowledge recursively. But I don't think that it's useful to compare this to ICL resulting from pretraining. For one thing whatever this algorithm is isn't buried in a deluge of other subject-specific priors. More importantly, the most powerful aspect of this algorithm is the fact that it can achieve sample-efficient distillation and generalization. This is exactly the part that needs to be tacked onto ICL with distillation over many rollouts!
Thanks for sharing the cellular automata paper! It's interesting, but I find the argument you're making to be unconvincing. If we take that paper at face value then what it suggests is that training on general syntactic rule-based data creates general priors related to downstream syntactic transmission i.e. induction heads. I don't dispute this, but the crucial advantage of human learning is the ability to build intricate semantic representations on the fly in a sample-efficient way. ICL makes use of specific pre-existing semantic representations, as well as at least somewhat general pre-existing syntactic representations, but I maintain that it cannot actually form new semantic structures. As a test case, I do not believe that you can remove all knowledge of e.g. quantum mechanics removed from the training data of an LLM, leave it with a quantum mechanics textbook and unlimited inference compute, and expect it to master quantum mechanics to the level of a capable human. I do not believe that you can do this even with in-context distillation or RL on quantum mechanics problems. I expect that this requires a vast amount of knowledge of quantum mechanics to be integrated into pretraining. This experiment is hard to run with commercial models, but it may be possible to do this with one of those vintage LLMs.
(Edit 2026-10-15: The main point I was trying to make is that discrete token language modeling is a very alien objective, and it is the only objective that has allowed us to achieve current intelligence levels with current levels of compute. World model/game based approaches that are more intuitively "human" in some ways have not yielded comparable results. Therefore discrete token language modeling may be very "data efficient" in some sense.
This is a comparison of ML objectives rather than architectures or algorithms. If you compare current ML architectures to humans on "data efficiency" for learning from language, ML loses sorely.
I think that trying to draw implications about models based on comparisons to humans is very tricky; this post intentionally does this at points for rhetoric, which may have been more trouble than it was worth.)
Humans are really bad at comparing themselves to models.
Take, for example, Leopold's preschooler graph:
Set aside, for a moment, concerns of scaling and RSI, and meditate: what characteristics does GPT-2 share with a preschooler?
I can't come up with anything else, because these two entities are almost completely disjoint. Is GPT-2 capable of bipedal locomotion, recognizing its mother's voice, or naming people by face? Does a preschooler learn from eight million scraped web pages sourced from Reddit?
What exactly is a preschooler "trained" on? Thousands of hours of "multimodal data". Assuming that a preschooler sees at 720p, and is awake for 12,000 hours by the age of 3 (about 11 hours a day), they have consumed at least 27 terabytes of video data at streaming-quality compression, or about 3.6 petabytes uncompressed, not to mention audio and sensorimotor data.[1]
In model-size terms, how large is a preschooler? Trillions of parameters, maybe. Beren Millidge's estimate, which assumes only ~1,000 synapses per neuron, puts the whole brain at an effective 10-30 trillion parameters.
So surely GPT-2 is much more "data efficient" than humans, wielding "preschooler"-level control of language with a measly 1.5 billion parameters and 40 gigabytes of text! And what about GPT-3, which, with only 175 billion parameters and about 1 terabyte of text, was capable of prose like:
Only a truly prodigious "elementary schooler" would be able to continue Janus' thought so eloquently. And yet this notion of LLM data inefficiency persists. "Frontier data efficiency lab" Flapping Airplanes, on their front page, writes:
Does this make any sense when you consider that most humans in the history of the species lived never "seeing" a single "text token" by adulthood, or in their entire lives? What do most human "samples" actually consist of? Yet a thesis like this is enough pretense for a $180M seed round and an estimated valuation of $5B.
Even OpenAI's Dan Selsam indulges this notion of data inefficiency, in an otherwise lucid tweet on AI risk:
It's natural that Dan, a capabilities researcher watching the marginal gains from text data go down, would say this. Also, nothing rules out enormous algorithmic advances that squeeze more out of the same quantities of data and compute.
But we really don't understand how humans scale, at all, and so it's absurd to claim that models are "data inefficient" compared to humans. At small scale, the discrete text token-based transformer architecture seems more efficient than anything we've ever seen.[2]
What about at large scale? As Dan concludes, as far as dangerous capabilities are concerned, it's efficient enough.
Yann LeCun does a similar calculation for a 4-year-old's optic nerve and lands at 10^14–10^15 bytes.
It's almost as if human language is a "fully general highly compressed mathematical isomorphism of human cognition". In that case Yann is also wrong, because language is the shortcut to human-level intelligence at current levels of compute; you can just slap robot capabilities and visual intuition on later.