It would be really bad if anyone solved continual learning. One of the big ways we lucked out with the LLM paradigm is that it takes months to train new models, which puts a limit on how fast a takeoff can go. Continual learning would ruin that.
It might be good to push for a ban on continual learning research. I'm guessing AI developers would resist this pretty hard, even though in principle they should support it—the "party line" is that they will figure out AI alignment as they go, and continual learning would make it way harder for them to do that.
I kinda think that continual learning is not that hard as a science problem, and its adoption is being slowed more by engineering constraints than anything else. (At least for the more simple forms of continual learning.) Basically, I would not be surprised at all if aicos are already working internally with non-toy models with some kind of continual learning mechanism. But updating such a model on a given set of information is likely to be much more expensive per token than context-reading or generation.
It might be sufficient to ban publications on continual learning. It would be much harder for an open-weights AI developer to fully solve the problem by itself, internally, with no help from academia or other AI developers.
4MichaelDickens
Another worry with continual learning:
A misaligned superhuman AI still has to solve the alignment problem, or else the next-gen AI will be misaligned to itself. A continual-learning superhuman AI only has to solve the value drift problem, which seems easier.