This is a special post for quick takes by Jan Betley. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
People often talk about continual learning as a fundamental unresolved problem on the path to the "real AGI".
To me, it feels like iterated distillation that is later put in context or accessed via tool calls (e.g. Claude Dreams) + maybe some simple process turning that into a finetuning dataset leading to a LoRA adapter that is activated when needed will quite likely be enough for any use case we might imagine.
My rough takes are like this:
* Model + scaffolding provides a good source of uplift compared to model only and this continues to be true
* Any uplift that comes via scaffolding can be in principle rolled directly into the model weights
My cruxes are:
* What are the capabilities advantages of keeping things in the scaffolding rather than pushing them into the weights? I could imagine that keeping things as tool calls is good for interpretability reasons. But I also could imagine that all else being equal, putting it in the weights makes it a more flexible / general capability.
* I also expect that there will be substantial leakage, i.e. capabilities are not cleanly described as (model-only) or (scaffolding-only), but rather will be split across both.