Continual learning might wake the world up to AGI, without yet bringing the dangers of AGI.
Pretraining gives shallow intelligence that is general, RL gives deep creative intelligence in a narrow skill, but it used to be very hard to make it work well for most skills. RL with pretrained models, which is RLVR, makes RL robustly applicable to a wide variety of narrow skills. But it still needs to be applied manually, the skills it trains are hand-picked before deployment, and so deep creative intelligence from RLVR remains jagged, compared to the more general shallow intelligence from pretraining.
Continual learning has now been vaguely announced for 2026 by both Anthropic and GDM. If it merely provides adaptation for AI instances to the current situation or job at the shallow level of pretraining, it might still be significantly more economically valuable than the current crop of models, leading to more visible job displacement, waking up the public and the politicians to more of the importance of AI. Yet it doesn't necessarily make direct progress in automating RLVR or introducing some other way of turning the creativity of RL general, and so the AIs won't necessarily get notably more dangerous than today at the level of disempowerment or extinction.
I disagree that it will 'wake the world up' in a more meaningful way than generic capabilities progress and broader adoption (i.e. 'no change in trend'), and my guess is that you're overestimating the AI literacy of the broader public and policymakers / how able they are to distinguish particular advancements from a general trend of progress.
I know a university professor ('Bob') who offered students extra credit if they:
1. Asked ChatGPT who the world's leading expert was in Bob's field.
2. Corrected it if it named anyone other than Bob.
3. Sent him a link to the chat log.
Not only did he think we'd unlocked continual learning; he thought that we'd unlocked continual learning across instances. This was in February 2024.
(maybe you're thinking 'oh, he just expected them to train on the chats later', but this is explicitly not true; he thought they were 'teaching' the model that he was the world's leading expert in his field, within that instance, and iterated on prompting techniques that he instructed his students to use, based on 'experiments' he ran)
There are similar stories involving policymakers and other powerful people that are easy to find. I just think the causal chain of people's awareness is only very vaguely related to specific advancements at all (e.g. reasoning models didn't make more news among 'normal people' than 'GPT 5', even though you could watch a machine think! which is surely an incredible headline).
3Vladimir_Nesov
I'm thinking of simply greater salience, compared to a more bearish trajectory with no continual learning (where chatbots are the new Google but people aren't losing jobs all over the place). If there are objective grounds for a public outcry, more people will pay more attention, including politicians. What they'll do with that attention is unclear, but I think continual learning has the potential for bringing significantly more attention in 2027-2028 compared to its absence, without yet an existential catastrophe or a straightforwardly destructive "warning shot".
I think I'm misunderstanding you, so I'm going to try and talk it through.
Ok, so there are three possible states for the near-future:
Advances in continual learning
Advances in non-continual learning capabilities
Progress stops
My impression is that neither of us think 3 is very likely. I think there are plenty of capability improvements that could lead to generic increased saliency (and that massive automation and economic upheaval is happening even if progress were to ~halt, which is the main 'generically increase saliency' lever). So continual learning does... (read more)