Agreed that the huge (in comparison to human short term memory) context size makes updating weights less pressing. Also agreed that training to improve the ability to write memories and documentation by looking at improvements in accuracy on subsequent tasks of a variety of types makes sense. I still think that weight updates have something to recommend them, notably the "grokking" phenomena where further training can reorganize a substantial body of weights.
I have a suggestion on incremental weight updating. I hope it is ok to place it here. If not, can anyone suggest a better location?
One idea for combating catastrophic forgetting that I'd like to add to the mix (alongside other work-in-progress in the field, like the Titans architecture, which addresses a different axis) is to retrain _relevant_ parts of the prior training along with the new continual/incremental training examples, which trickle in sparsely, one at a time.
This is similar to MIR (R. Aljundi et al. NeurIPS 2019 https://arxiv.org/abs/1908.04742) but with a somewhat different problem case and a different choice of trade-off.
This would be a more selective version of rehearsal/replay.
To be more specific, suppose that we have a pre-trained model (I'll talk about adding some data to pre-training shortly) and we've been given a single new example to train on. We may want to crank up the learning rate (since we have one datum, not umpteen) and do the usual gradient descent (more on this later). But we don't want to forget previously learned skills!
Can we pick out what is most at risk and try to preserve it? Well, the new datum is going to drive some parameter updates. Say we consider the N parameters with the largest changes from the new datum as the riskiest places.
If we could pick out the _old_ training examples most relevant to these N parameters we could retrain with those _old_ training examples as well as with the new datum. We could hopefully converge to parameters that preserve the old skill and yet add the new one, finding weights consistent with _both_.
How might we locate these old training examples? Well, during pre-training there are parameter updates too. For each parameter, there is some pre-training example that produced the maximum update to that parameter. Keep that example during pre-training. More precisely, if we keep the pre-training examples in some known, persistent, order, just keep the _index_ of the pre-training example that produced the maximum update for that parameter and the magnitude of that update. One can extend this by keeping the k-highest update indices.
This is affordable, costing a few bytes per parameter (optionally multiplied by k) for the pointer and the magnitude, a few terabytes on disk for a teraparameter model.
The pre-training compute cost is also small, a numerical compare per parameter per learning step, O(P) cost, the same order as the optimizer's own state update. These compares only rarely trigger a record write - about k*ln(|corpus|)/|corpus| of the time.
One probably wants to track the update normalized by Adam's second-moment estimate, typically already maintained per parameter, to ameliorate scale differences across parameters and scale changes during training.
One caveat is that this keeps the magnitude of the update at the time that the model is trained on the example, so this magnitude is stale at the time the new example is examined. However, one wouldn't want to try to remedy this by recalculating the parameter change due to the old example with the current model, because the current model has already learned from the old example, so a recalculation of the parameter change due to the old example with the current model would understate the parameter learning the old example caused. A well-absorbed sample has a small gradient at the current weights precisely because it was absorbed.
Note that persistent pre-training data is a different problem definition from MIR's, so this task doesn't compel reservoir sampling like MIR's problem definition does. If the incoming data is sparse, e.g. emails from a project, it is reasonable to store them persistently as well, so the equivalent of MIR's "M" set is the whole corpus - but, as in MIR, budgeting compute is still important.
The main difference from MIR is that MIR's calculation of the incremental loss imposed on the old training example is exact for the foreseen update, at the cost of an expensive calculation requiring two forward passes through the model per candidate examined. This cost forces MIR to score only a random subset "C" of "M" - which suffices for a small buffer but degrades to near-zero recall of at-risk examples as "M" grows to corpus size.
The per-parameter index replaces 'score a random subset and hope' with a direct lookup of the historically implicated examples.
Here, the entire corpus is considered, with O(1) cost to discover each interfered old example, at the cost of approximating the incremental loss by looking just at the most perturbed parameters. In other words, by approximating using the leading term by parameter coordinate of the dot product of the old and new samples' parameter updates, the hopefully dominant term being the term for the parameter used to retrieve the old sample.
In the spirit of the _signed_ loss calculation in MIR, and selection of their replay set "B" from samples with the largest _increase_ in loss, rather than the largest unsigned change in loss: Since we are concerned with _interference_, we really want to replay old training examples which moved parameters in the _opposite_ direction from the new, sparse training example.
So we should keep _two_ k-highest update indices, one for parameter increases and one for parameter decreases, replaying those with their sign opposite to the new sample's parameter update. Since examples with the same sign as the new one still carry some risk, e.g. of parameter overshoot, one might replay all k examples from the opposite sign indices and the top one or few from the same sign examples.
A second option to potentially adopt from MIR is using multiple iterations, epoch-ettes, instead of, or in addition to, increased learning rates to emphasize the new sample, together with the retrieved old samples. Two advantages of this are that one can revisit the choice of _which_ old samples to use on each iteration and one follows the nonlinear components of the gradient in parameter space. The disadvantage is that computation cost rises linearly with iteration count, albeit in the current idea the cost of _finding_ each old sample is O(1), while MIR needs 2 forward passes per sample in the "C" set.
To compare with TracIn (Pruthi et al., NeurIPS 2020 https://arxiv.org/abs/2002.08484), TracIn finds training examples which are responsible for large loss changes on a subsequent test case - its best-known application is surfacing mislabeled data. The current idea is intended for cases where fully correct training examples e.g. suggested an overly general deduction. E.g. if, in training, all vehicles happened to be cars and had 4 wheels, and the training led to deducing vehicle->4 wheels, but the new datum is an 18 wheel truck, the deduction can be modified to car->4 wheels, accommodating both the old and new data. TracIn also requires revisiting the whole training corpus, while the current idea retrieves a small set by the parameters a new example perturbs most.
This could be tested on a small model and data set: train on the data set, find or construct an example (like the 18-wheel truck) whose incorporation prompts forgetting, then train on that example and compare forgetting under no replay, random replay, and indexed replay, with equal replay budgets for the latter two.
Thanks to Claude (Anthropic) for extensive discussion — literature pointers, critique of earlier drafts, and several refinements noted in the text.
What do you think?
Or: When Memories Get Good -- The Default Path Without Theoretical Breakthroughs
Epistemic status: Fairly confident in the core thesis (context + memory can substitute for weight updates for most practical purposes). The RL training loop is a sketch, not a tested proposal. I haven't done a thorough literature review.
Suppose there are no major breakthroughs in continual learning -- that is, suppose we continue to struggle at using information gathered at runtime to update the weights of a given instance of an AI model. If you try to update the weights at runtime today, usually you end up with catastrophic forgetting, or you find you can only make very small updates with the tiny amount of useful data you have [1] .
So, if you can’t train a day’s worth of information into the model, how could you end up with something that functions as if it were learning on the job?
Long Context Lengths, High Quality Summaries, and Detailed Documentation [2] [3] .
It’s a straightforward idea, and basically done today, just not particularly well yet. Laying it out:
That’s it.
Why Doesn’t This Work Now?
Firstly -- it kind of does. In my own software projects I maintain a concise Claude.md file (which gets passed to each new agent on spawn), as well as extensive documentation which the Claude.md points to (and which the Claudes can search at will). Claude and ChatGPT already produce and store ‘memories’ in this way through their existing harnesses. These work okay, and we know that models can effectively learn in context.
But it doesn’t work that well yet. I suspect this is because current models just aren’t very good at writing or at using these notes.
It’s actually a very hard task. We’re basically having the model ask itself “What do I know that a fresh instance doesn’t, that would be useful for it to remember across all future instantiations?” and then asking it to write this down using as few tokens as it possibly can.
For a model to be able to do a good job, it needs to understand whether the things it knows are coming from its current context or its weights, and accurately guess how a future instance will respond to the memories. Basically, it needs to have a good theory of mind.
I think the difficulty of this task is the main reason memory especially sucked when it first came out. There are plenty of examples of irrelevant memories being created inside ChatGPT, for example.
It also took some time to train models which understood what the memories were and how to use them. Previously, models would attend too strongly to memories in irrelevant contexts, bringing up notes where they don’t belong. Kimi K2.5 still struggles with this, in my experience, seeing notes at the start of its context window as very important and relevant, even in situations where they shouldn’t be.
Claude ignores the apple note. Kimi always finds a way to bring it up.
But memory is getting much better, and newer models use it more successfully. I expect that as models get more intelligent their use of memory and documentation will continue to improve, especially in the world where this is trained for explicitly. Models are also getting better at handling the retrieval of dense information across their long context windows, so a mundane prediction that these trends continue should point us towards prosaic “continual learning” becoming quite useful over 2026 and 2027.
It also should be noted that memories like this are functionally the same as compaction (summaries written by the AI when reaching the end of the context window, so it can continue working). In both cases the model is writing compressed information to pass to a future instance to (hopefully) perform better. This is already an optimisation target for frontier labs.
How We Could Make It Work Better
We can easily train models to create and use memory as an RL task. To sketch out a simple method -- suppose that when finishing a task, instead of scoring the model’s performance immediately, we have the model write memories and documentation, and then we run a new instance on the same, similar, and dissimilar tasks [5] with those memories and documentation, and have a reward function which scores on the combined performance (with some small penalty for the length of the memories). This looks like:
The reward function used for the actual parameter updates would be a function of the scores across each of the models, plus some penalty relative to the length of the memories and the total context length of the model.
There are several other ways to do something like this, of course, and some would be much more efficient than what I have laid out here. I’m mainly trying to get across a few key ideas:
Overall I would expect this to reward both the model’s ability to write AND to understand its memories and documentation, with some risk of pushing the model towards very dense, difficult to read memories (ala linguistic drift).
I haven’t spun up an experiment to test this empirically, but may do at some point. If anybody else would like to, or has done so already, please let me know!
Could This Replace Real Continual Learning? What About Intelligence Gains From Having The Information In The Weights?
There are two things going on here that we need to untangle. The first is about the model having the correct information to achieve its goals. This is what gets put into the memories and the documentation, and what is addressed by prosaic continual learning.
The second thing we wonder about is how to increase the intelligence of the model. How can it do more with less information, or figure out new things that it wasn’t told, or get better at acting in the world in a general sense.
With prosaic continual learning, the real intelligence gains only happen in the next generation of AI models.
Suppose Claude 5 is launched with a 1m context window, and it is smart enough to write good [8] documentation and memories. If a task uses about 500k of context, and produces about 1000 tokens of new memories, then doing ten tasks a day, every day, you can run the model for 50 days before you hit the ceiling on how many memories you can store [9] [10] .
Then, 50 or so days later, Claude 5.1 is launched, with improved capability by the usual process. Claude 5.1 inherits the existing memories and documentation and immediately works on improving and compressing them [11] . Combined with a longer context window, the new Claude 5.1 might buy another 50 days of memory [12] .
Repeat ad nauseam, or at least until Claude N solves true continual learning with parameter updates at runtime.
In this way, the lessons from a particular deployment (say, by a model that has been answering phones for a particular company) are trivially passed from one generation to the next while capabilities continue to improve via regular training. In practice, is there anything more we need true continual learning to do? [13]
Can We Have A Human Brain Analogy, Please?
One of the reasons continual learning is so popular a concept is because humans do it, which makes it a very attractive answer to the question “What can’t AI’s do yet?”.
The human learning process looks something like the above chart, where we have an explicit, discrete, and extremely small working memory, which holds somewhere on the order of 10 objects in memory at a time. This probably exists as activations in the pre-frontal cortex. It’s analogous to LLM’s context window, being lossless and explicit, but is far, far smaller.
Then, humans have a kind of buffer, where information is stored on the order of hours to weeks in a lossy but easily accessible way. This seems to be held in the hippocampus. You can draw a weak parallel to AI reading documentation here, being some partially processed summary of what has happened, accessible with a few seconds of thought.
Humans can read documentation too, of course, but the read speed is extremely slow in comparison. AI is able to read documentation at a speed that is more comparable to a human recalling a specific memory.
Next, humans have long term memory, which is slowly updated on the order of days, probably by reading and updating against the hippocampus’ “buffer” [14] . This is where the missing piece for LLM’s continual learning would be an analogy, if we knew how to properly update an instances’ parameters at runtime.
Finally, even humans don’t become more intelligent after reaching full adulthood [15] . We rely on evolutionary selection to make any significant changes to human intelligence. The analogy here is to the next generation of AI models being trained, although that happens far, far faster.
Laying it out like this, you can see the ‘long term memory’ update step is missing, but the ‘context window + documentation’ is ridiculously larger in storage capacity that human working and short-term memory, and the ‘intelligence gain’ step so much shorter, that skipping a weight update at runtime might be viable. Humans require memory related parameter updates because we can’t store much information in working or short term memory, but if our working memory was so large it didn’t fill up within our lifetimes, you can see how the situation changes.
Conclusions
Having now thought through this, I have updated away from continual learning being a real issue for AI capabilities in the near future [16] .
It doesn’t seem like it is needed for general purpose capability improvements, where the regime of releasing a new model every few months works fine.
It doesn’t seem like it’s needed for company specific work, where you can store all of the needed information in documentation and in context.
I think the fact that it has to be written and used explicitly by the models is a satisfying answer to why it hasn’t worked well so far -- the models simply haven’t been smart enough to do a good job at this so far.
I’m also bullish on progress on this problem being fast, given that this performance is something that can be straightforwardly optimised with unsupervised RL, including training models to handle and edit stale memories.
Overall... damn, I guess we’re making continual learners now.
People think about the goal of continual learning as being ‘the model can learn on the job’, so, practically speaking, the main use case is for specific, non-generalisable data unique to this deployment of the model. When I say you don’t have enough data to do this usefully then, I mean, one days’ (or one months’) recording of work is a tiny amount of data to try to fine-tune a model on. You can’t reliably learn new things this way, though you might be able to elicit existing knowledge in the model. ↩︎
This is not a new idea. Dario spoke about it on Dwarkesh, and a quick Claude search reveals several different papers talking about the concept, most of which I haven’t read in detail. I am writing this post because I haven’t seen it clearly, publicly combined in one place before, and maybe there’s some interesting exploration of the RL training loop and why explicit memory has been a hard thing for models to get right. ↩︎
We also have versions of all of this today, which is why it’s “prosaic” continual learning. ↩︎
You could also include things like tools the model has built for itself, information it's found online and wants to make a note about, and really anything that is created or curated for the models’ use without the entire thing being stored in the active context window. ↩︎
Same task means literally the exact same task [17] .
Similar task means tasks pulled from the same narrow distribution. For example, the set of things a particular employee might do in their work for a single company. We want to encourage memories that are useful across this somewhat narrow domain.
Dissimilar task means tasks pulled from more radically different distributions. Coding, psychological support, creative fiction, etc. I think we need to include some probability of dissimilar tasks in the batch in order to train the model to not rely too strongly on memory. At deployment time, the model may indeed be given memories that are irrelevant for the task at hand.
If I had to take a random stab at the proportion of each type of task assigned for a given batch, I would weight the distribution so that the N+1th task is about 89% likely to be from the similar distribution, 10% likely to from the dissimilar distribution, and 1% likely to be the exact same task repeated. ↩︎
The model should write memories and documentation on both successful and unsuccessful attempts at the problem -- it likely has useful information about what to try or not to try either way. I’m also imagining that there is some penalty for overall token usage when training for inference efficiency reasons -- that would incentivise the passing of useful tips and lessons via memories and documentation, if it can make the later instances more efficient.
It is even fine to pass the entire solution via memory, so long as the model has learnt when it doesn’t apply, and has been suitably penalised for the memory length. I think we can get this result by tuning the proportion of same, similar, and dissimilar tasks being scored together -- that is, if we run similar tasks n times, and dissimilar tasks m times (and possibly the same task p times), with the memory and documentation passed through for a given reward calculation, we can select n, m, and p such that generally useful tips are favoured over long and specific instructions. ↩︎
I’m unsure whether documentation should be length penalised or not. You get this to some degree by measuring the performance of the model using the documentation. I’d lean towards probably not, using the principle of allowing the training to choose whether short or long documentation is better. I’m assuming we use a tool which allows the model to choose to read some reasonable amount of tokens at a time, rather than risk breaking things by dumping entire files in, or only clipping them when they become very long. ↩︎
In the memory case, ‘good’ means that they can figure out what would be useful to know in all future runs, and can recover from bad or missing memories by editing it later. In the documentation case, it means they can include all the relevant information accurately, avoid including slop, and then use the information to be much more effective than they would be without it. ↩︎
I made up numbers here just to show how much room there is. In this case, I get 1 million token context window, minus 500k task buffer, leaving 500k tokens for memories. At 10,000 tokens per day, we get 50 days of memory buffer.
This is also kind of a ‘worse case scenario’. A thousand tokens for memories for each task is very high, since most memories could simply be pointers to where the real detail lives, and you would quickly run out of new things to write. Do you memorize ~6000 words worth of new information every day, and keep it memorised for the rest of your life? If you can compress your new memories to only 1000 new tokens per day instead of 10,000, you get over a year of runtime. Alternatively, increasing context length from the current 1m tokens also provides wiggle room. ↩︎
Different tasks will have very different profiles here. For example, coding might require only very short memories, whereas piloting a robot through a factory might require memories that include a map and descriptions of every mistake the model had made on previous trips. ↩︎
We can expect a new version to be better at the difficult task of creating and using the memories & documentation, especially if it’s trained explicitly for this. Some possibilities here, which point towards shorter and fewer memories:
- Is the model able to figure out what a memory is trying to say more easily, letting it compress existing memories into fewer tokens?
- Is the model better at writing the information more densely?
- Does the model know more information intrinsically, allowing it to remove that information from its explicit context?
- Is the model better at knowing what it knows, allowing it to cut unnecessary memories?
- How does the new version change the tradeoff between memory and documentation caching? E.g. is it faster at reading documentation? Is it better at knowing when and which documents to read given its particular goals?
↩︎I am pretty confident that memory usage should be able to grow slow enough that a Claude working for a particular company can fit everything it needs into context and explicit documentation. For this not to be the case, you have to assume that extremely large amounts of information are needed (multiple books worth), and that you discover new information that must be held in context (rather than in documentation you can look up) at a rate faster than the context window grows, and that future models won’t be able to significantly compress existing memories or be able to move existing memories into documentation by virtue of being better at knowing when to look up things. ↩︎
In the limit, this process is functionally identical to continual learning, as far as I can tell. Just imagine the 50 days between model releases reducing to some short period, like a day or an hour, and imagine the written memories that are passed forward becoming denser and denser, an abstract initialisation pattern that is loaded in for a deployment (like a static image).
Putting the same scenario the reverse way, imagine a model with traditional, weight-updating continual learning. Rather than updating its weights directly, it (like humans) uses a short term memory buffer to store new information and isolate private information from the weights. Every hour, the relevant lessons from the previous hour’s work are trained into a copy of the model, which is then seamlessly switched out, and the buffer updated. ↩︎
I don’t know if you’ve ever noticed your long term memory updating, I feel like I have. Have you ever had a major event happen, and then only some days later have cemented a behaviour change, even though you knew the change was necessary from the moment of the event? ↩︎
They continue to learn more, which makes their crystalised intelligence (knowledge and skills) go up, but their fluid intelligence (ability to reason abstractly, solve new problems, etc) declines after early adulthood. ↩︎
I’m even coming around on continual learning being worse for most mundane uses -- suppose you have your own version of a model, with the weights updated to store information specific to you and your use case. What happens when a new model is released? You have to retrain? What happens to the optimisations from batching? ↩︎
I actually think it’s debatable whether you should include the literal same task as an option for the nth instance (with the memories and documentation prepared by the (n-1th) instance) to be assigned. If you do this, the model could just include the whole solution in its memories, but honestly, for some production usage and types of task, that could be a reasonable and viable strategy.
I think in general we should try to train on the same distribution as the deployment, so whether to include the literal same task (vs just similar tasks) as a possible option here depends on whether you think that’s a situation that is likely to occur in practice (maybe setting up the same programming environment many times?), and whether you get anything from doing this (quickly using the cached procedure?). ↩︎