I agree with the first part of your comment, but about "in-context learning" - this is just really shallow to the point that I don't believe it's worth comparing to a bona fide "learning algorithm" at all. It's just what transformers do; every time they generate a new token, influences are propagated from prior generated tokens according to pathways learned by the model during training. If you provide the model a list of instructions, the attention mechanism will impart that information into later latent vectors, which condition the output tokens in such a way that they are likely to obey those instructions. But that only works because these signals slot into a specific pre-existing conceptual model encoded into the weights that is sufficient to interpret those instructions and retrieve them at precisely the right times. When a human learns how to navigate the world, or masters a discipline, or even works on a specific long-horizon project, they build new conceptual models that enable them to organize and use information effectively. LLM in-context learning does not achieve this. It resembles what we do when we recollect what we had for lunch, or consult an instruction manual. The only ways to build deeper knowledge into an LLM are pretraining and to a lesser extent, RL. Both of these are tremendously less efficient for LLMS than for humans, even when there's no reason humans should have any kind of evolutionary prior (for example the data efficiency for learning math or code).
I also disagree with the analogy between pretraining and evolution. Our DNA just doesn't contain that much information - less than 2 gigs! Almost of all that has no relevance for the brain at all. There's not really any room in that to customize how our brains function beyond overall structure and cell types. Certainly there's no way that could wire in any approximation to our faculties, which modern neural nets need billions of custom set parameters to emulate. That stuff has to come from learning after birth.
These concepts are discussed eloquently in many places by Steven Byrnes (who comments elsewhere on this page), and in this post by Dwarkesh.
(Edit 2026-10-15: The main point I was trying to make is that discrete token language modeling is a very alien objective, and it is the only objective that has allowed us to achieve current intelligence levels with current levels of compute. World model/game based approaches that are more intuitively "human" in some ways have not yielded comparable results. Therefore discrete token language modeling may be very "data efficient" in some sense.
This is a comparison of ML objectives rather than architectures or algorithms. If you compare current ML architectures to humans on "data efficiency" for learning from language, ML loses sorely.
I think that trying to draw implications about models based on comparisons to humans is very tricky; this post intentionally does this at points for rhetoric, which may have been more trouble than it was worth.)
Humans are really bad at comparing themselves to models.
Take, for example, Leopold's preschooler graph:
Set aside, for a moment, concerns of scaling and RSI, and meditate: what characteristics does GPT-2 share with a preschooler?
I can't come up with anything else, because these two entities are almost completely disjoint. Is GPT-2 capable of bipedal locomotion, recognizing its mother's voice, or naming people by face? Does a preschooler learn from eight million scraped web pages sourced from Reddit?
What exactly is a preschooler "trained" on? Thousands of hours of "multimodal data". Assuming that a preschooler sees at 720p, and is awake for 12,000 hours by the age of 3 (about 11 hours a day), they have consumed at least 27 terabytes of video data at streaming-quality compression, or about 3.6 petabytes uncompressed, not to mention audio and sensorimotor data.[1]
In model-size terms, how large is a preschooler? Trillions of parameters, maybe. Beren Millidge's estimate, which assumes only ~1,000 synapses per neuron, puts the whole brain at an effective 10-30 trillion parameters.
So surely GPT-2 is much more "data efficient" than humans, wielding "preschooler"-level control of language with a measly 1.5 billion parameters and 40 gigabytes of text! And what about GPT-3, which, with only 175 billion parameters and about 1 terabyte of text, was capable of prose like:
Only a truly prodigious "elementary schooler" would be able to continue Janus' thought so eloquently. And yet this notion of LLM data inefficiency persists. "Frontier data efficiency lab" Flapping Airplanes, on their front page, writes:
Does this make any sense when you consider that most humans in the history of the species lived never "seeing" a single "text token" by adulthood, or in their entire lives? What do most human "samples" actually consist of? Yet a thesis like this is enough pretense for a $180M seed round and an estimated valuation of $5B.
Even OpenAI's Dan Selsam indulges this notion of data inefficiency, in an otherwise lucid tweet on AI risk:
It's natural that Dan, a capabilities researcher watching the marginal gains from text data go down, would say this. Also, nothing rules out enormous algorithmic advances that squeeze more out of the same quantities of data and compute.
But we really don't understand how humans scale, at all, and so it's absurd to claim that models are "data inefficient" compared to humans. At small scale, the discrete text token-based transformer architecture seems more efficient than anything we've ever seen.[2]
What about at large scale? As Dan concludes, as far as dangerous capabilities are concerned, it's efficient enough.
Yann LeCun does a similar calculation for a 4-year-old's optic nerve and lands at 10^14–10^15 bytes.
It's almost as if human language is a "fully general highly compressed mathematical isomorphism of human cognition". In that case Yann is also wrong, because language is the shortcut to human-level intelligence at current levels of compute; you can just slap robot capabilities and visual intuition on later.