Next word prediction RLVR is massively more compute hungry per unit of data (than pretraining), so it's both likely impractical at current levels of compute, and plausibly solves text data scarcity at 2028-2030 levels of compute if it's useful. The benefit is generality of the objective, the same as with pretraining itself, compared to manual construction of RL environments for narrow tasks. Given pretraining vs. RLVR capability gap, it's plausibly a big deal if it makes RL-level capabilities as general as the current shallow pretraining-level capabilities.
The fact that Łukasz Kaiser (transformer paper co-author, currently at OpenAI) is talking about it in Nov 2025 is strong evidence AI companies couldn't yet rule out that it might work. The idea itself is obvious enough, but that's less significant as evidence for its prospects.
The point is the distinction between pretraining and RL (in the level of capability), and between manual jagged RLVR and hypothetical general RL (in the generality of capability). I think observing Opus 4.5 and Gemini 3 Pro is sufficient to be somewhat confident that even at 2026 compute levels pretraining itself won't be sufficient for AGI (it won't train sufficiently competent in-context learning behavior to let AIs work around all their hobblings), while IMO gold medal results (even with DeepSeek-V3 model size) demonstrate that RLVR is strong enough to get superhuman capabilities in the narrow skills it gets applied to (especially when it's 1-4T active param models rather than DeepSeek-V3). So in the current regime (until 2029-2031, when yet another level of compute becomes available) AGI requires some kind of general RL, and continual learning doesn't necessarily enable it on its own, even if it becomes very useful for the purposes of on-the-job training of AI instances.
This is more of a claim that timelines don't get shorter within the 2026-2028 window because of continual learning, even if it's understood as something that significantly increases AI adoption and secures funding for 5+ GW training systems by 2028-2030 (as well as rouses the public via job displacement). That is, starting with timelines without continual learning (appearing as its own thing, rather than an aspect of AGI), where AGI doesn't appear in 2026-2028, I think adding continual learning (on its own) doesn't obviously give AGI either, assuming continual learning is not actual automated RLVR (AIs applying RLVR automatically to add new skills to themselves). After 2029-2031, there are new things that could be attempted, such as next word prediction RLVR, and enough time will pass that new ideas might get ready, so I'm only talking very near term.
Continual learning might wake the world up to AGI, without yet bringing the dangers of AGI.
Pretraining gives shallow intelligence that is general, RL gives deep creative intelligence in a narrow skill, but it used to be very hard to make it work well for most skills. RL with pretrained models, which is RLVR, makes RL robustly applicable to a wide variety of narrow skills. But it still needs to be applied manually, the skills it trains are hand-picked before deployment, and so deep creative intelligence from RLVR remains jagged, compared to the more general shallow intelligence from pretraining.
Continual learning has now been vaguely announced for 2026 by both Anthropic and GDM. If it merely provides adaptation for AI instances to the current situation or job at the shallow level of pretraining, it might still be significantly more economically valuable than the current crop of models, leading to more visible job displacement, waking up the public and the politicians to more of the importance of AI. Yet it doesn't necessarily make direct progress in automating RLVR or introducing some other way of turning the creativity of RL general, and so the AIs won't necessarily get notably more dangerous than today at the level of disempowerment or extinction.
P[doom] ... it makes sense for individuals to spend most of their time not worrying about it as long as it is bounded away from 1
That has no bearing on whether we'll be OK. Beliefs are for describing reality, whether they are useful or actionable doesn't matter to what they should say. "You will be OK" is a claim of fact, and the post mostly discusses things that are not about this fact being true or false. Perhaps "You shouldn't spend too much time worrying" or "You should feel OK" captures the intent of this post, but this is a plan of action, something entirely different from the claim of fact that "You will be OK", both in content and in the kind of thing it is (plan vs. belief), in the role it should play in clear reasoning.
1if anything, it seems more common that people dig into incorrect beliefs because of a sense of adversity against others
Consider cults (including milder things like weird "alternative" health advice groups etc.). Positivity and mutual support seem like a key element of their architecture, and adversity often primarily comes from peers rather than an outgroup. I'm not talking about isolated beliefs, content and motivations for those tend to be far more legible. A lot of belief memeplexes have either too few followers or aren't distinct enough from all the other nonsense to be explicitly labeled as cults or ideologies, or to be organized, but you generally can't argue their members out of alignment with the group (on relevant beliefs, considered altogether).
the point ... is to make it clear that when you are receiving kindness, you are not receiving updates towards truth
This is also a standard piece of anti-epistemic machinery of groups that reinforce some nonsense memplex among themselves with support and positivity. Support and positivity are great, but directing them to systematically taboo correctness-fixing activity is what I'm gesturing at, the sort of "kindness" that by its intent and nature tends to trade off against correctness.
Some links on modal logic for FDT-style decision theory and coordination:
once you achieve pareto optimality, there is a tradeoff between kindness and correctness
It's hard to stay on a pareto frontier, optimizing for more (or less) "kindness" directly is a goodharting hazard. If you ask for something, you might just get poisoned with more of the fake version of it.
I'd prefer less of the sort of "kindness" that trades off with correctness, rather than more of it (even when getting less of it wouldn't actually help with correctness; it just doesn't seem like a good thing). But if I ask for that, I'll end up getting some (subtle) sneering and trolling, or unproductive high-standards elitism that on general principle wants to destroy ideas that didn't get a chance to grow up yet. Similarly, if you ask for the sort of "kindness" that does trade off with correctness, you'll end up getting some sycophancy (essentially) that cultivates your errors, making them stronger and more entrenched in your identity, ever more painful and less feasible to eventually defeat (even if there are benign forms of this sort of "kindness" that merely don't make the problem worse in a comfortable way, as opposed to trying to intervene on it).
having an identity is an important part of how nearly everyone navigates this complex and confusing world
Legible ideas (that are practical to meaningfully argue about) cover a lot of ground, they are not as hazardous as part of identity. And less well-defined but useful/promising/interesting understandings don't need to become part of identity to be taken seriously and developed. That's the failure mode at the other extreme, when anything insufficiently scientific/empirical/legible/etc. gets thrown out with the bathwater.
rather than immediately coming in with a wrecking ball and demolishing emotionally load bearing pillars
Probably when something is easy to defeat (admits argument, legible), it's not that painful to let it go. The pain is the nebulous attachment fighting for influence, that it won't be fully defeated even when you end up consciously endorsing a change of mind. Thus ideologies are somewhat infeasible to change, they'll keep their hold even long after the host disavows them. A habit of keeping such things at a distance benefits from other people not feeding their structurally hazardous placement (as emotionally load bearing pillars) with positivity. But that's distinct from viewing positively the development of even such hazardous things, handling them with appropriate caution.
it can be deeply emotionally painful to part ways with deeply held beliefs
This is not necessarily the case, not for everyone. Theories and their credences don't need to be cherished to be developed, or acted upon, they only need to be taken seriously. Plausibly this can be mitigated by keeping identity small, accepting only more legible things in the role of "beliefs" that can have this sort of psychological effect (so that they can be defeated through argument alone). Legible ideas cover a surprising amount of territory, there is no pragmatic need to treat anything else as "beliefs" in this sense, all the other things can remain ambient epistemic content detached from who you are. When more nebulous worldviews become part of one's identity, they become nearly impossible to dislodge (and possibly painful, with enough context and effort). They are still worth developing towards eventual legibility, and not practical to argue with (or properly explain).
Thus arguing legible beliefs should by their nature be less intrusive than arguing nebulous worldviews. And perhaps nebulous worldviews should be argued against being held as "beliefs" in the emotional sense in general, regardless of their apparent correctness, as a matter of epistemic hygiene. Ensuring by habit you are not going to be in the position where you have "beliefs" that would be painful to part ways with, and also can't be pinned down clearly enough to dispel.
I'm thinking of simply greater salience, compared to a more bearish trajectory with no continual learning (where chatbots are the new Google but people aren't losing jobs all over the place). If there are objective grounds for a public outcry, more people will pay more attention, including politicians. What they'll do with that attention is unclear, but I think continual learning has the potential for bringing significantly more attention in 2027-2028 compared to its absence, without yet an existential catastrophe or a straightforwardly destructive "warning shot".