If you've ever screamed in all-caps at an AI, then you know the difference between what it learned when it was trained, and what you can teach it by prompting. The LLMs powering today's AI don't learn on the job the way people do. They learn all at once in a big training run and once that's done, we freeze the parameters that store their skills and knowledge.

So we all get the same AI with the same skills and biases, centrally trained by a frontier model company. Beyond turning everything even more same-y, there's an economic cost to this centralization: firms use AI that lacks understanding of their unique rules, culture, and quirks. Humans learn this "tacit knowledge" on the job through observation and (sometimes painful) feedback, but AI with its frozen parameters cannot. With context engineering, we can augment the prompt to help AI remember facts, but not teach it skills that last.

It's not surprising that the domains where AI is most successful, like coding, are those suited to centralized training. Good software development skills are mostly firm-agnostic. For everything else firm-specific, AI is trained to trawl the codebase and build context from scratch for every single task. Human developers don't do this. It would be absurd to block off a week for "establishing context" before starting every task. Human developers develop durable conceptual models of the codebase over time. From-scratch context is a band-aid that works for AI coding because codebases are a good source for up-to-date context (with some exceptions). Since AI doesn't learn on the job (a capability called continual learning), we need an analog for the "codebase" before we can replicate AI's success with coding in other domains. But no such analog exists universally.

In 1945, the economist Friedrich Hayek described what became known as "Hayek's knowledge problem", which states that critical economic information needed to run firms efficiently is dispersed and local, and not available centrally. To this day, it remains a core argument in favor of markets, and against central economic planning. So it's wild that some mainstream economists now argue that AI makes central economic planning more likely^[1], in part by making tacit knowledge more codifiable. This twist aside, more codified tacit knowledge doesn't solve Hayek's problem because without continual learning, AI won't be up to date with it.

For all of these reasons, AI that continually learns tacit knowledge will be far more economically valuable than today's AI. But the effect won't be uniform for every application of AI. One of the most promising and popular such applications, (AI remains otherwise deeply unpopular) is accelerating science for the discovery of new therapies and materials. For that, continual learning isn't optional. No training data (human-generated or synthetic) covers what's off the edge of the scientific map. Hundreds of millions of dollars are flowing into companies that are building scientific loops around AI, but unless it's able to continually learn from experiments like human scientists, how far could it really go beyond what's already known?

Xerox Builds an Expert System AI

In contrast to today's AI, humans learn continually but it takes time. For example, when moving between firms within their own discipline, humans take years to get back to their original productivity.^[2][3] So companies have an incentive to find ways of accelerating the process. One such company was Xerox. In the early 1980s, it was struggling to recruit enough ex-military technicians to service its machines in the wild, and wanted to lower the skill bar for new technicians.

The result was documentation in the form of decision trees that guided a technician from symptoms to diagnosis. The trees were built by sampling and diagnosing faults in the lab. A decade later, AI researchers building expert systems at Xerox's PARC sought to use their models to replace the decision trees.^[4] They built a system called RAPPER for a single module of the machines, and demoed it to the technicians. When asked whether it would be useful if scaled up to cover the whole machine, this is what they said:

"Not really – though it is amazing – rather like a bear dancing."

RAPPER was based on an engineering model of the module and did not learn over time from the technician's experience. RAPPER was faithful to the decision trees in the documentation (it covered every fault they did), but the hard problems were new ones not in the documentation. Looking for insights, researchers rode along with the technicians on service jobs. What they found were heuristics, stories swapped at work-group meetings, and a radio network which technicians used to solicit ideas from buddies if they were stumped. They got an idea:

It suggested to us that we could stand the artificial intelligence approach on its head, so to speak; the work community itself could become the expert system

RAPPER couldn't learn over time, but the researchers could build a tool to make it easier for the technicians to learn from each other. The tip economy of the technicians became the blueprint for the system that eventually saw adoption: Eureka was not an AI, but rather a searchable database of validated tips collected by the field engineers and technicians. These tips were the definition of codified, tacit knowledge: technical and specialized to Xerox. For example:

Diagnosing unusual, costly failures—Bimetallic corrosion builds up on A and causes intermittent failures that seem to be B. Replacing B makes the problem seem to go away because A is moved in installation. First clean A, and later replace by new gold-plated AA, available as Part #1234...

Eureka added value to Xerox because the hard problems were new and few technicians knew how to fix them. In such a data-poor environment, the technicians could learn from each other with Eureka, rather than from just their own direct experiences.

Tacit Knowledge Has Few Samples, and LLMs Need Many

Xerox is not unique in needing employees to learn specialized tacit knowledge to contribute. Such knowledge is specific to a firm or even team, proprietary, and mostly not written down. Some of it, like the knowledge the Xerox technicians learned about diagnosing and fixing machines, can't easily be written down at all. It needs to be learned from a few examples, or "samples".

Humans (and animals) are good at learning from a few samples. There's likely evolutionary selection at play, especially for learning harmful associations. For example, rats that are given sweet water then injected with a drug that makes them sick learn negative associations from only a single experience,^[5] which makes sense: If you eat a berry and get sick, it's useful to your survival to never eat another one. Even if you got sick from something else, it's better to miss out on a few tasty berries than eat a poisoned one.

The LLMs powering AI in comparison are not sample-efficient during training, lagging somewhere between 3 and 6 orders of magnitude behind humans,^[6][7] which is a problem if you want them to learn a firm's tacit knowledge from a handful of samples in the wild.

But wait, can't we teach LLMs by adding a few examples of what we want to the prompt? Yes, however, there's no way to commit what the LLM learns from the prompt samples (a process called "in-context learning") to the LLM's parameters. So the examples stay in the prompt. But the prompt has finite length. If we naively push new observations into it, old ones will have to drop off, and we'll lose what's learned from them. Maybe we could solve the length issue by curating just the right samples into the prompt for each task? We'd store them in a database, then grow that with new observations. The prompt stays open for just the samples relevant to the current task. But observations will keep coming in, and eventually the total samples relevant to a task won't fit again. We'll need a way to "compress" the samples into an abstract form. And the abstraction should generalize to novel tasks. And wait isn't that exactly what LLM training is for? It is, and it's why we can't use in-context learning continually. The extra pieces we need to make it work, roughly map to what LLM training already does, and we don't know how to do that continually!

Humans are Incredibly Sample-Efficient

So any continual learning needs to be stored in the LLM parameters, which leaves us with a gap of 3 to 6 orders of magnitude in sample efficiency. How do we overcome this gap? For comparison, it took ~20 years for the cost of genome sequencing to move by ~5 orders of magnitude. And the No Free Lunch theorem tells us there's no hope for a magic algorithm that universally bridges the gap and can learn anything we throw at it.^[8] Improvement will only come from buying efficiency at the cost of generality, using inductive priors in the learning algorithm that exploit some structure in the training data.

But humans are general learners! How do we do it? Evolution has baked inductive priors into our learning algorithm that are matched to our environment.^[9] We're not actually general learners, just good at learning most of what we encounter. Remember the sick rats? They didn't form the same distaste when the experiment was performed with "bright-noisy" water instead of sweet water. Some inductive prior triggered by "sweet yet harmful" ramped up the learning rate for that experience, but not when the water was switched out. The mechanism is selective and tuned by the machinery of evolution.

Sample efficiency and inductive priors go hand-in-hand. I wrote about how AlphaFold used specialized inductive priors to learn how to fold proteins from the relatively small protein database. Its priors were static and designed by specialists just for protein folding. Humans have analogous static priors that, for example, make us attend to (and learn from) snake and spider images faster than the modern equivalent categories like guns or electrical outlets. But static priors for avoiding dangerous animals we evolved with don't help a modern human learn how to operate an iPad with only a few demonstrations. And yet human sample efficiency extends well past tasks that evolution could optimize for, from operating iPads to abstract math. Whatever inductive prior machinery powers this, it must be adaptive enough to extend efficiency to tasks our evolution could never prepare for.

Is Sample Efficiency All You Need?

Let's assume we can bestow the magic recipe that makes human sample efficiency on LLMs. Could we then continue training forever, and update the parameters of a user or firm's dedicated LLM as new observations roll in?

It turns out that the differences between humans and LLMs don't stop at inductive priors and sample efficiency. If we continued training an LLM, even a sample-efficient one, it would eventually lose plasticity^[10] and stop learning anything at all. Even worse, new observations would start to eat what the LLM learned from the old ones well before loss of plasticity kicks in. It's called catastrophic interference,^[11] and the standard mitigation is to randomize the order of the training data. If the data is random, no concept or topic is bunched up at a single point during training to be wiped out by later data. Here's an excerpt from a real training batch (a group of sequences the LLM trains on simultaneously):

Learning like this would be as if your kindergarten curriculum were made up of quantum mechanics, the history of woodworking, and the alphabet, all taught in random order.

In a continual learning setting, we don't get to pick the order of the observations. If we ignore randomization and just continue training with new observations as they come, we'd interfere with and degrade what the LLM already learned. If we fight this catastrophic interference by shuffling the handful of new observations in with the massive original training data, the tiny volume of new observations would be completely overwhelmed. The LLM would learn very little from them.

But let's just assume we solve both sample efficiency and catastrophic interference. Would we then have a model that learns continually from a user or firm, picking up all the tacit knowledge today's LLMs lack?

Yes, but it's worth pointing out a few practical challenges with the premise. Frontier LLMs have parameters in the trillions, and moving them about in datacenters takes time and power. The latest Kimi K3 flagship release has 2.8 trillion of them. Freezing them after training means one copy can serve everyone. Your Claude prompt gets grouped together with others into a batch and the GPU re-uses the same frozen parameters when generating outputs for all of them. One startup is even etching the parameters directly into silicon.

Ignoring all of this and assuming we can figure out an efficient way to serve personalized parameters to every user, we could continue to train this idealized model on new observations and solve continual learning, just like in humans. So how do we solve catastrophic interference, and how do we make model training sample-efficient?

Learning to Walk Before You Run

How did humans solve continual learning? Does that tell us anything about how to solve it for LLMs? We've seen that humans owe their sample efficiency to inductive priors. But unlike AlphaFold whose specialized and static priors give it efficiency in a narrow domain, humans are generally sample-efficient. For example, a surgeon learns the tacit knowledge of their field from their first-hand experience in the role.

How would inductive priors for surgery even work? We can be sure there aren't any encoded in the human genome. One clue is in the difference between how LLMs and humans need their training data ordered. LLMs train on randomly ordered data to prevent catastrophic interference. Humans in contrast need strict ordering to learn the prerequisites first. There's no way to learn surgery without first learning anatomy. If you try, the anatomical parts of it will be incomprehensible. If we view anatomy as a learned inductive prior for surgery, we could imagine how humans keep their sample efficiency at the expense of needing concepts to be learned in their compositional order, from simple to complex.

There's evidence of compositional learning of priors in animals, albeit for simpler tasks than surgery. Rats (again) that had already learned the "schemas" for flavor-place association tasks could learn new pairs in a single trial, a process that would otherwise take weeks.^[12]

"systems consolidation can occur extremely quickly if an associative 'schema' into which new information is incorporated has previously been created… New traces, trained for only one trial, then became assimilated and rapidly hippocampal-independent."

So why don't we do this with LLMs? Why not learn concepts in order, and have the LLM bootstrap more complex concepts by using their prerequisites as inductive priors? Because catastrophic interference wipes out earlier concepts when training the more complex ones.

If we solved catastrophic interference, would existing transformer-based LLM architectures automatically use prerequisites as priors to efficiently learn complex concepts? The answer is maybe. We at least know that training LLMs with a curated curriculum of topics ordered compositionally does not work as well as training them with data in random order.^[7] The deficiency could be simply a matter of solving catastrophic interference, but it's unclear whether further problems await beyond that. I could not find an experiment that disentangles the two and uncovers whether LLMs use past knowledge as inductive priors for future training. My guess is that the two problems have different shapes and require different solutions, but I hope I'm wrong.

Beyond Centralized Training

Xerox's attempt to build an AI to help their technicians diagnose machine faults failed in part because it didn't learn alongside them. What the technicians needed was help with new problems, not AI trained on the spec and documentation they already had. What worked with 90s-era technology was not an AI at all, but a system for sharing tips to improve the technicians' own continual learning. Xerox's problem persists today: the tacit knowledge necessary to contribute to firms is unique, generally not written down, and must be learned from a few samples. Only humans can do this right now.

We live in the centralized AI training era. LLMs are sample-inefficient and difficult to train, so training is done once, carefully, and with massive curated datasets. But the economy is vast and to contribute to its many firms you need tacit, firm-specific knowledge that's not written down, and needs to be learned from just a few examples. For AI's impact on science, it needs to learn from new experiments to go beyond existing science, just like humans would. It can't do this right now and the problems barring it are research-shaped, but humans and their efficient continual learning give us some hints about what the solutions could look like.

1. Brynjolfsson & Hitzig, AI's Use of Knowledge in Society.

   ^{^}

2. Boris Groysberg, Linda-Eling Lee & Ashish Nanda, Can They Take It With Them? The Portability of Star Knowledge Workers' Performance, Management Science 54(7), 2008, 1213–1230. Extended in Chasing Stars (Princeton University Press, 2010).

   ^{^}

3. Matthew Bidwell, Paying More to Get Less: The Effects of External Hiring versus Internal Mobility, Administrative Science Quarterly 56(3), 2011, 369–407.

   ^{^}

4. Daniel G. Bobrow & Jack Whalen, Community Knowledge Sharing in Practice: The Eureka Story, Reflections (SoL Journal) 4(2), 2002. academia.edu.

   ^{^}

5. John Garcia & Robert A. Koelling, Relation of Cue to Consequence in Avoidance Learning, Psychonomic Science 4, 1966, 123–124. DOI:10.3758/BF03342209.

   ^{^}

6. Michael C. Frank, Bridging the Data Gap Between Children and Large Language Models, Trends in Cognitive Sciences 27(11), 2023, 990–992. DOI:10.1016/j.tics.2023.08.007.

   ^{^}

7. Alex Warstadt et al., Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora, CoNLL 2023, 1–34. babylm.github.io.

   ^{^}

8. David H. Wolpert, The Lack of A Priori Distinctions Between Learning Algorithms, Neural Computation 8(7), 1996, 1341–1390. DOI:10.1162/neco.1996.8.7.1341.

   ^{^}

9. Elizabeth S. Spelke & Katherine D. Kinzler, Core Knowledge, Developmental Science 10(1), 2007, 89–96. DOI:10.1111/j.1467-7687.2007.00569.x.

   ^{^}

10. Baekrok Shin, Junsoo Oh, Hanseul Cho & Chulhee Yun, DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity, NeurIPS 2024. arXiv:2410.23495.

    ^{^}

11. Timothée Lesort, Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes, PhD thesis, 2020. arXiv:2007.00487.

    ^{^}

12. Dorothy Tse et al., Schemas and Memory Consolidation, Science 316(5821), 2007, 76–82. DOI:10.1126/science.1135935.

    ^{^}