LLMs not scaling to superintelligence is insufficient, otherwise just being 100x faster would let them do conceptual/theoretical research (that doesn't require expensive slow-world feedback) 100x faster, which plausibly feeds into speeding up AI research and making a better use of hardware, to make them actually superhumanly smarter on top of being 100x faster.
So a premise like a "lack of data efficiency" is indeed necessary. But this specific premise doesn't fully work with RLVR (in a broad sense that allows graders to also invoke LLMs, carefully dodging reward hacking attempts), where we now have multiple glimpses into how world class human performance can be developed or exceeded using a reasonable amount of training data defining tasks/environments/graders. LLM scale will plausibly still make this much easier by the time the near term potential for brute scaling of LLMs runs out, both on the side of the LLMs getting easier to train, and on the side of LLMs being ready to automate the whole process, performing it with their own hands and no human involvement.
I think this still doesn't lead to superintelligence on a predictable schedule, so that running through the remainder of LLM scaling doesn't imply a high probability of LLM-enabled superintelligence. But the reason for that is slow learning loops when using RLVR in the current paradigm, rather than "lack of data efficiency", the learning of deep skills in LLMs will without new breakthroughs happen at the speed of model releases (weeks to months) rather than at the speed of token generation (100x human speed). This doesn't stop LLMs from going RSI (fully automating the process of training the next model and making the kinds of improvements humans would be able to make), but it's a slow kind of RSI that doesn't lead to a takeoff.
Crucially, this doesn't dodge the ASI danger, merely uncouples the technical origins and timing of the danger from the LLMs themselves. The danger is still much higher today than 10 years ago because of all the datacenters and better scale-up systems being built with the LLM money, making it much easier to experiment or to very quickly scale successful experiments to the level of a global catastrophe. LLMs would also probably be directly involved as major ingredients of such hypothetical ASI, or able to eventually invent ASI themselves. There just doesn't seem to be a strong reason to expect that they will make such invention much easier or faster (because of the slow learning thing) than it would've been without their labor, if all the datacenters and scale-up systems were still magically available, and the LLM technology was there as a possible basis for the improvements towards ASI (without itself participating in the work).
Yes I see that you agree that RSI doesn't lead to a sudden takeoff. I also see that you can make lots of relevant data post training with RLVR. That would sure help, but I am not sure about helping the AI make the kind of blue-sky conceptual breakthroughs that geniuses make occasionally. For some things there simply isn't enough data even in principle that could be created. For example, "create a new branch of mathematics" - I don't think LLM's could ever do.
You are right that inference speed is >> than model release, and that ties to something related and perhaps unavoidably linked to data efficiency. Biology also learns with far less FLOPs than ANN as well as needing far less data. I am assuming that any data efficient architecture will also require far less training FLOPS, also solving the slow model release problem.
See 1 Jan 2026, 1 Jan 2025 and the July 2025 update. This continues my habit of documenting my beliefs and feelings as we transition to ASI.
1. LLM and Transformer progress
We have seen significant progress since my last update of course, with Mythos and similar. I still believe that transformers won't get to TAI but I believe its now more likely that they will speed up the search for new architectures. I believe that the more capable transformers are without being TAI, the better - as we can adapt and learn from them. My timelines are 2-10 years, but with more probability on the shorter end because transformer LLM's will speed up the discovery of a better paradigm. For example its more likely there will be less or no plateau because transformers will find the next architecture, or greatly speed up the process of finding it. I will say 50% that Tx finds the better architecture by or before 2030 now.
I said before
This is perhaps my most unusual claim, that is even if an LLM could just beat the smartest person on the planet at relevant AI research tasks, that does not mean it necessarily could self improve all the way its hardware would allow.
I'd still give this about 50% chance of being true, however I am less confident now.
Like before, I believe data efficiency is the major point lacking, and scaling LLM to 1GW data centers won't substantially change that.
2. What if there isn't a much better architecture, how slow could take-off realistically go?
I have been considering that more, specifically the case where the more data efficient architecture and system (human learning in congenitally blind people is far more data efficient than LLM training) is not more parameter efficient. We know that biology is more sample/data efficient but there is no evidence I am aware of that it is more parameter efficient for a similar well trained model. In some cases well trained transformers seem to require fewer parameters for similar performance.
Specifically in our situation, lets say that with limiting data centers to ~1GW, around 100T models are possible, but not too much more. We get a transformer model at that size, learn a lot from it, but its not fully TAI. The new architecture would let us train such a model faster and with less data, but not make it impossibly smarter. Additionally, the training method that biology uses is not likely to match our current GPU hardware that well, one of the reason the transformer is successful is that the algorithm fits the hardware.
With the new architecture, its likely that we could train larger models, say 1,000 T because of the greater data efficiency, meaning we need less data and training time. However such a model may not be impossibly more capable than the 100T transformer one.
In this case, we get "mild" superintelligence, but not something beyond our understanding. Additionally it comes just at the time when Moore's law is almost finished and diminishing returns as I discussed here and you get a very different world to what many here assume. There is nothing in this outcome that would surprise me, and as far as I am aware, its not ruled out by existing results.
Many people who are in tech, but not AI experts believe something like this and their actions make sense if things turn out this way. For example, if the current HW is not x-risk, then politics and freedom become far more important. Sharing the secrets of AI far and wide asap is seen then as a fight against central tyranny and essential and good, rather than dangerous. Many on twitter genuinely believe this. For them MIRI is very misguided and intent on taking away essential freedom under the guise of AI safety.
3. Takeover risk
Mine is lower than many here.
4. If you are so smart, then
why are you not richfind interesting things for meOnce on x.com I came across this someone promoting this result
I didn't know this, and found it surprising. I intuitively expected you could take a massive mirror, and fry a small point to much hotter than the suns surface. But then it because clear it was obviously true and hence important because my intuition was wrong.
This got me thinking, with my spare codex usage, can GPT do similar things. I gave it this example and challenged it to come up with more. After all it has almost the entire scientific literature somewhat memorized!
However it failed repeatably. It first came up with 10 supposedly similar examples, however they were much simpler and I got the correct answers. It tried then a few more times with 10 or so potential examples and was not able to come up with something simple and similar to this where my intuition was incorrect. I then went a stage further, and said something similar to "just surprise me with something interesting" and it still was not that impressive.
I then got thinking, when are an AI's "type 1/intuitive" vs "type 2/thinking" answers different? If they are, what does it mean?
An obvious potential answer is that RLHF or other post training introduces bias, towards politically correct interpretations rather than scientific ones.
I have not got codex on a schedule using my tokens investigating the scientific literature for contradictions or surprising results where some parts of the establishment contradict others. Nothing spectacular so far, but will keep it going. I told it not to do AI related stuff as it would then flag pretty much all long terms forecasts as suspect and we already know that.
5. I think the Orthogonality Thesis is misguided and harmful
I came across discussion of the OT/OH again (Lumpenspace etc) and it has renewed and increased my belief that it is either irrelevant or incorrect and actively harmful in general truthseeking and beliefs in the relevant area.
Consider the situation where the OT was never invented, and we started with a view like:
Intelligence requires a world model, and goals are viewed, assessed, judged through a world model.
That is, a simple cell can have a "world model" and a goal to swim up the gradient of a useful chemical. It can't have a goal of "improve humanity" because it has no concept of it. The concept cannot fit in its world model so it cannot be a goal.
A superintelligent AI with a model of our world cannot have a goal of "swim to a better environment" because what does that even mean. It can however have a goal of "improve humanity" or "take the universe for itself". This does not make the super AI any more likely to be safe or dangerous but such a framing would have avoided many misunderstandings among the technical but not specialized public. The OT is almost universally misunderstood, either taking to say more than it does and hence "disproven" (intelligent people commit less crime so OT is false) or quoted as a fact about current and forthcoming AI systems, in a way that is totally unjustified.
I believe that there is status quo bias here, where the OT persists instead of being abandoned. If you disagree, consider the situation/alternate timeline where a different, more practical framing took hold. I don't expect the OT would then displace it and become popular. It would never get beyond a footnote. If you believe the OT wouldn't displace it, then why is it popularized in this situation.
To me, thinking about world models and intelligence, and coming up with the OT and random mind spaces has an analogy in thermodynamics. The major takeaway from the 2nd law is not that there is some chance for all gas particles in a container to randomly be in one half of the container purely for random reasons, yet we cannot rule it out. This is about the most irrelevant and useless but still true conclusion you can draw from a model of particles randomly hitting each other!
The OT's focus on the total mind space, instead of what we can expect from AI that are trained in the real world seems just as misguided. The OT is also very often a Motte-and-bailey fallacy where people swap between what seems like the strong and weak versions in the middle of a discussion. For example it can start as a claim that the AI's we will produce will be randomly mis-aligned because of the OT, but then retreat to the "all possible minds" position.