Continual learning is likely a key precursor to both lack of monitorability and superintelligence. So from the policy perspective, intent to develop continual learning should be treated similarly to intent to develop superintelligence, or intent to abandon faithful CoTs and monitorability.

When continual learning goes even a little bit beyond preparing directories with Markdown files, and at least starts to help with comprehension of contexts with unbounded length, this results in the thinking of the AIs gaining an unbounded opaque serial depth. Such AIs would either have opaque recurrent data (essentially temporary weights/parameters being trained at test time; but not using a human-written optimizer), or directly update (some of) their original weights. Their later reasoning will then be based on this opaque data rather than only on the text of the context in front of a given instance of the AI. The recurrent data (or updated weights) is not an interpretable bottleneck, but rather a cumulative opaque representation of all previous test-time training (within some session, whose length could be much higher than a few million tokens of a single context). This recurrent data could represent hidden plans and attitudes and intentions that don't get reflected in any external behavior or routinely interpretable data, and only occasionally and intentionally determine the precise effects that the AI decides to enact. Few things currently motivate abandoning upper bounds on opaque serial depth like enabling continual learning does.

When continual learning is more ambitious and does the kind of RLVR-grade learning that enabled the 2025-2026 LLMs to go from being confused about whether 9.11 is greater than 9.9 to solving a Millennium Prize problem, this probably leads to rapid accumulation of conceptual inventions, a key thing current LLMs are conspicuously bad at. Right now, new incremental ideas only get baked into the LLMs with new model releases, after they get into the training data. So even though small incremental ideas can be invented many orders of magnitude faster, at token-generation rather than at model-release speed, the overall invention-learning-invention loop is still very slow, because of the slow learning step that is an integral part of the loop, and that step only advances with new models.

If conceptual inventions are serially deep collections of small incremental ideas, then all that's missing in the current LLM/RL paradigm is a fast invention-learning-invention loop. Strong continual learning would make learning fast at test time, making this loop go fast, and that's probably sufficient to enable conceptual invention, accumulation of deep technological culture that quickly reaches far-future technologies (at least to the extent that the experimental feedback doesn't constrain), including architecture-rewriting strong RSI and superintelligence.