Not being useful is often not an important problem, for meanings of "useful" that are not tautologically referring back to the stated purpose of the activity itself (like you can do with chess). Usefulness is an occasional side effect and a meaningful indicator for many activities, but targeting it directly can be disruptive. Short feedback loops are great where they can be found, which isn't everywhere.

When the purpose of an activity isn't clear, "usefulness" in the tautological sense doesn't really work, and playing more is a central thing that helps with finding out more about what this purpose should naturally be. Being too early in stating purposes too clearly results in goodharting to some arbitrary indicator. And "usefulness" in some other non-tautological sense will generally just ask you to go do something else (which won't be about playing chess).

The egoism/altruism distinction makes sense in the framing of boundaries and scopes of optimization for different preferences. If you optimize things that are centrally yours (within the boundary of what is you or yours) according to values of egoism, and optimize things that are centrally not yours (outside this boundary) but within some circle of concern according to values of altruism, this is a healthy state of affairs. (Enemies could form yet another scope of optimization, with values of coercion and such.)

This works for interventions that either mostly affect things on one side of the boundary or the other, but breaks down when they significantly affect both, when there are tradeoffs. An important failure mode (perhaps downstream of having to deal with situations that have tradeoffs) is when values appropriate for one scope of optimization get directly applied to another, when egoism or coersion target your circle of concern, or altruism gets turned back on yourself (or enemies). Or when the boundaries get redrawn in a strange way (someone with a lot of power could consider quite a lot of other people "theirs", applying egoism, optimizing for effectiveness without respect for autonomy).

Kinda, but there won't be enough natural text data at the higher end of this range (using 2028-2030 compute) to just scale pretraining on text (you'd need more than 1,000 trillion tokens with repetition, maybe 200-300 trillion unique tokens), something else would need to happen instead or you start losing efficiency and compute ends up being less useful than it would be if there was enough text.

The steps of scaling take a long time, so only late 2025 models get to be shaped compute optimally for 2024 levels of pretraining compute, and run on hardware announced and first available in the cloud in 2024. This is just 2 years from 2022, when GPT-4 was trained, and the first of two 10x-20x steps at the 2022-2026 pace of scaling, with a third step remaining somewhere beyond 2026 if we assume $100bn per year revenue for an AI company (at that time). With 2026 compute, there just might be enough text data (with repetition) to say that scaling of pretraining is still happening in a straightforward sense, which brings the change from original Mar 2023 GPT-4 to 100x-400x (for models that might come out in 2027).

But this 100x-400x is also a confusing point of comparison, since between 2023 and 2027 there was the introduction of RLVR scaling (and test time reasoning), and also all the improvements that come from working on a product (as opposed to a research prototype) for 4 years. Continual learning might be another change complicating this comparison that happens before 2027 (meaning it might be a significant change, which remains uncertain; that it's coming in some form, at least as effective context extension, seems quite clear at this point).

"Culmination" suggests a subsequent decline. In 2025, scaling of RLVR delivered a lot of capabilities, and late 2025 was the first time since 2023 that 10x-20x scaling in pretraining compute (compared to original Mar 2023 GPT-4) has finally made a full appearance (probably in Gemini 3 Pro and Opus 4.5). There is 100x-400x more in scaling of compute by 2029-2031 (compared to the current models), and much more in the low-hanging fruit of doing things well rather than prototyping the first thing that sort of works. The only low-hanging fruit that likely mostly ran out in 2025 is raw scaling of RLVR (in proportion to pretraining), and even that probably still has a bit to go. Setting up better tasks and RL environments will plausibly be more impactful than some feasible amount of further scaling of RLVR relative to pretraining. Then there's continual learning that might be quite impactful in 2026-2027.

I expect some sort of culmination in 2027-2032 (assuming no AGI), when scaling of compute slows down and there have been at least 1-2 years to learn to make better use of it. The first stage of compute slowdown probably follows 2026, if 2028 won't see 5 GW training systems (which would be on trend for training system compute growth in 2022-2026, but currently doesn't seem to be happening). The second probably follows 2028-2030, when funding to secure ever more training compute mostly stops growing, and so compute mostly falls back to growth according to its price-performance.

I did attempt to preempt this kind of response with "some guardrails on effective and ethical use of those galaxies" in my comment. There exists a reasonable middle ground where individual people get significant say on what happens with some allotment of resources, more say than the rest of humanity does. Disagreements on values have a natural solution in establishing scopes of optimization and boundaries between them, even if these are not absolute boundaries (let alone physical boundaries), rather than mixing everything together and denying meaningful individual agency to everyone.

The reasonable question is which portions should get how much weight from individual agency, and which portions must be shared, governed jointly with others. But starting from the outset with little shared resources and (obviously) allowing establishment of shared projects using their resources by agreement between the stakeholders doesn't seem much different from some de novo process of establishing such shared projects with no direct involvement of individuals, if the stakeholders would indeed on reflection prefer to establish such projects and cede resources to them.

(I'm of course imagining a superintelligent level of agency/governance, to match the scale of resources. But if humans are to survive at all and get more than a single server rack of resources, ability to grow up seems like the most basic thing, and governing a galaxy at the level of a base human does seem like failing at effective use. Preferences extrapolated by some external superintelligence seem like a reasonable framing for temporary delegation, but these preferences would fail to remain legitimate if the person claimed as their originator grows up and asks for something different. So ultimately individual agency should have greater authority than any abstraction of preference, provided the individual had the opportunity to get their act sufficiently together.)

Within the hypothetical of ASIs that won't just largely ignore humanity (leading to its demise or permanent disempowerment), there is a claim about distribution of resources according to AI company stock. A lot of the post argues with the hypothetical rather than the claim-within-the-hypothetical, which as a rhetorical move is friction for discussing hypotheticals. This move ends up attempting to prove more than just issues with the claim, while ignoring the claim, even as it's not its intent.

(The claim isn't ignored in other parts of the post, but what seems wrong with the framing of the post is the parts that are about the hypothetical rather than the claim. To illustrate this, I've attempted to defend coherence of the hypothetical, importance of the issue of distribution of resources within it, and the framing of individual humans ending up with galaxy-scale resources.)

Dogs pee on trees to mark their territory; humans don't respect that. Humans have contracts; ASIs won't respect those either.

There are billions of reachable galaxies, and billions of humans. Different people will develop different values, and property rights are just what it means for each person to have the freedom to pursue such values. The form of property rights will be different, AI company stock might not survive as an anchor for distribution of the cosmic wealth, and some guardrails on effective and ethical use of those galaxies likely make sense. Shares in future compute probably make more sense as units of property than actual physical galaxies. But individual humans ending up with galaxy-scale resources is a reasonable way of operationalizing the outcome where ASIs didn't permanently disempower the future of humanity.

On current trajectory, ASIs seem likely to largely or fully ignore the future of humanity, but possible alternatives to this are not as a whole category simply meaningless, a sign of ignorance and lack of imagination. Splitting the physical resources becomes necessary once the world is past the current regime of seemingly unbounded growth and runs into the limits of accelerating expansion of the universe. For example, there is probably a tradeoff between running a lot of things in parallel, which would then run out of useful compute relatively early, and stockpiling matter for fewer things to survive into much deeper time, long after the galaxy clusters can no longer communicate with each other. There might be disagreements about which way to go with this between different people.

If you only update weights in LoRAs, they can be like KV-caches, different for each request and taking up only a reasonable amount of memory, preserving all the benefits of batching and keeping the users isolated from each other.

I'm thinking of simply greater salience, compared to a more bearish trajectory with no continual learning (where chatbots are the new Google but people aren't losing jobs all over the place). If there are objective grounds for a public outcry, more people will pay more attention, including politicians. What they'll do with that attention is unclear, but I think continual learning has the potential for bringing significantly more attention in 2027-2028 compared to its absence, without yet an existential catastrophe or a straightforwardly destructive "warning shot".

Next word prediction RLVR is massively more compute hungry per unit of data (than pretraining), so it's both likely impractical at current levels of compute, and plausibly solves text data scarcity at 2028-2030 levels of compute if it's useful. The benefit is generality of the objective, the same as with pretraining itself, compared to manual construction of RL environments for narrow tasks. Given pretraining vs. RLVR capability gap, it's plausibly a big deal if it makes RL-level capabilities as general as the current shallow pretraining-level capabilities.

The fact that Łukasz Kaiser (transformer paper co-author, currently at OpenAI) is talking about it in Nov 2025 is strong evidence AI companies couldn't yet rule out that it might work. The idea itself is obvious enough, but that's less significant as evidence for its prospects.