accumulation of crystallized intelligence that was produced by their own fluid intelligence
Essentially this isn't happening right now. I think it very likely starts happening within a few years, but will be slow, maybe slower than humanity, and as a result won't be able to quickly fix the problem of being slow.
"actual" AGI—the kind that probably doesn't already exist—the kind that has fluid intelligence and AI advantages for recursive self-improvement, which together make it likely to take over the world shortly after being created
Humanity should count as actual AGI, but it remains slow. Similarly, I think LLMs that do prosaic RSI of automatically building the next model (crucially including formulation of new RL tasks/environments/graders) will be actual AGI in the sense that they can (on their own, without humanity's input) eventually generate and accumulate as much crystallized intelligence as humanity would.
Yet there is a bottleneck of the speed at which they learn novel deep skills (generate and crystallize new pieces of intelligence), going from new RL tasks to new models to new teaching moments that inspire new tasks for the new models that are ready to use the opportunity. This keeps the process at the slow pace of building models rather than at the fast pace of generating tokens, and so AI advantages don't quickly snowball into fast RSI that changes the fundamental nature of the AI.
Eventually fast RSI happens, but it plausibly takes many years of slow-learning prosaic RSI (for it) to figure out how to make that happen, and humans could end up being faster at figuring it out. As with any basic research, there is no trend that meaningfully predicts how many years it takes. All this time, there's actual AGI of slow-learning model building that can figure out anything eventually, but only slowly. This AGI might be the lesser factor of danger in triggering fast RSI, compared to the vast amount of compute its usefulness finances, which enables ambitious experiments and rapid scaling of prototypes.
We have large gaps in performance between AIs and humans (sample complexity for learning, ability to generate novel good concepts)
Pretraining very sample inefficiently reconstructs cognitive skills that left evidence about their nature in the text. RL training can use relatively short formulations of tasks/environments/graders to generate even novel pieces of cognitive skills needed to solve the tasks (that the authors of the tasks didn't necessarily have, as they didn't necessarily know how to solve the tasks, or how to do so efficiently). In this way, RL training is both sample efficient (with respect to the short formulations of tasks, and the tiny insights each task promotes), and a way of producing novel pieces of intelligence that can be crystallized in the new models.
What's currently missing is automated formulation of new RL tasks/environments/graders in response to gaps in existing crystallized intelligence (in the current model) with respect to the situations/problems it comes in contact with. I estimate that LLMs of 2031 will be more capable than Mythos 5 by about 3x as much as Mythos 5 is more capable than Opus 4.5+ (strictly before Opus 5). This is likely a gap notably wider than the Sonnet-Mythos gap, above the current frontier models that are already in a weight class capable of producing strong technical results and spontaneous competent cyberoffense. Taking that further step to the models of 2031 is probably insufficient for leaps of insight that quickly show how to make RSI go fast. But this is very likely enough for a base model of 2031 to be sufficiently teachable (using RL tasks) to end up learning how to formulate new RL tasks/environments/graders on its own, starting the slow-learning prosaic RSI process of building the next model automatically.
1
Introduction
I think reprogenetics (human germline genomic engineering) can be done in a widely acceptable and beneficial way, and should be pursued aggressively. In particular, as a strong background motivation of mine, I think accelerating strong reprogenetics is probably the best way to enable strong human intelligence amplification; and I think strong HIA is among the best ways to decrease existential risk from AGI.
A very common objection to caring much about reprogenetics is that AGI seems very likely to come soon—say, within a decade or two. (Here I mean "actual" AGI—the kind that probably doesn't already exist—the kind that has fluid intelligence and AI advantages for recursive self-improvement, which together make it likely to take over the world shortly after being created.) The objection is fairly straightforward:
Now, of course this is true to an extent. If AGI comes within 15 years, reprogenetics is almost totally useless. To the extent you believe that will happen, you believe reprogenetics is quantitatively less useful in expectation. Also, having a faster HIA method would of course be great.
However, I believe that this line of reasoning has led to a very mistaken underallocation of funding, talent, and other resources towards human intelligence amplification in general, and reprogenetics in particular. So, I would like to push back in a few ways:
(Note that this isn't a comprehensive fair-and-balanced report, but rather a collection of arguments in one direction. For example, I won't here discuss reasons that HIA could increase existential risk from AGI. Also, some but not all of these arguments rely on the assumption that alignment is very difficult.)
HIA, part of your nutritionally complete portfolio
HIA is super neglected. Would you rather be the 3000th person working on AI safety, the 300th person working on AI regulation, or the 3rd person working on accelerating HIA?
There's broad philanthropic alpha in working on HIA. It's tractable (to accelerate), neglected, and important. In particular, because it's somewhat taboo (though probably less than you think), there's relatively low-hanging fieldbuilding fruit to pick.
We must attend to all plausible points of intervention.
Uncoordinated resource allocation leads to unbalanced portfolios.
Against confident short timelines
I don't in fact think that confident short timelines (say, ">80% within 15 years" or similar) make much sense. As yet, I have not heard a clear and convincing case that AI research has already uncovered the engines that would produce a smarter-than-human general intelligence. The main arguments for AGI coming soon boil down, I think, to the rapid rise in capabilities and to the use of AI in AI research.
Regarding the rapid rise:
We have large gaps in performance between AIs and humans (sample complexity for learning, ability to generate novel good concepts).
Also, we have an apparent explanation for the rapid rise in capability: hoovering up big piles of internet data for quasi-imitation. Current AIs are skillful / knowledgeable approximately when there's a bunch of human training data for that thing. Of course, you need more elements if you want to explain all of the rapid rise, such as unhobbling via harnesses and RL, as well as computer advantages (scaled RLVR, speed, parallelism, copying, inexpensiveness).
But the core explanation of "the bulk of these capabilities come from human demonstration" holds water, as far as I know. In particular, this explanation says that current AIs got their capabilities via some route other than accumulation of crystallized intelligence that was produced by their own fluid intelligence. Namely, they got their capabilities via "copying" (broadly construed) crystallized human intelligence originally generated by human fluid intelligence.
This seems to largely explain away the rapid rise in capabilities, without invoking current AIs having much fluid intelligence. Combined with the apparent gap, with humans still well ahead on general fluid intelligence, I don't see how anyone gets to being very confident that we already have most or all of the ideas that would be needed for AGI.
Regarding the use of AI in AI research:
I expect the most important kinds of research to not be accelerated much, because they are not bottlenecked by coding in the first place. Unless you already think we're close to making AGI for other reasons, so that what's left is mainly the kind of research that is greatly accelerated by current or near-future AI, this partial acceleration shouldn't change your timelines much.
For previous discussion, see "Do confident short timelines make sense?" and "Views on when AGI comes and on strategy to reduce existential risk". I'm open to debating this with anyone who'd like to make a serious, public case for confident short timelines.
HIA may indirectly slow down AGI capabilities
The main stated justification for pursuing AGI capabilities is that AI / AGI would bring abundance for humanity. It stands to reason that if there were a credible, workable plan to get the (supposed) benefits of AGI without the huge existential risk (and other disempowerment of humans), then there would be less motive to develop AGI and it would be harder to justify pursuing AI capabilities. (See "5.1. Abundance makes less motive to make AGI".)
I don't know how large this effect would be. Presumably not very large. Presumably many people trying to increase AI capabilities just want money, power, status, or other selfish things, and would find some other justification. I'm not sure though; it's hard to put upper or lower bounds on the importance of underlying spiritual currents in society of hope, motivation, the longer-term future, and so on.
HIA has substantial impact even with short timelines
Approximate summary of this section:
In dire situations, if you're trying to be strategic about winning, it's advisable to play to your outs. That is, avoid the temptation to focus on incremental tractable gains that don't actually increase the chances of overall success. Instead, aim at paths to overall success, including by setting yourself up to take advantage of such opportunities, even if any specific path is unlikely.
Working towards a pause / stop / ban on AGI progress is a top priority. But what is a pause for? We pause, and then what? How do we more robustly prevent existential risk? Hopefully, we can ban Red AGI research, but probably not Blue AGI research progress. (Though social / political pressure might possibly be able to significantly slow down even Blue research.) In the longer run, how do we avoid making unaligned AGI? HIA is a way to improve our long-run chances by giving humanity more brainpower to find good answers to that question.
Suppose you do get a pause on AGI progress, but you don't have a plan for how to more robustly stop existential risk. More specifically, suppose that progress is not being made towards the outs. Then the pause is being wasted: for the moment you're bailing water out of the boat as it leaks in, but at some point more and more leaks will be sprung. You have to also be patching the hull. Ten years from now, when you have hopefully updated toward slightly longer timelines, you'll regret not getting started on the longer-term solutions back in 2026. I know I regret not working on this a decade ago, rather than bashing my head against the AGI alignment problem. HIA will make some progress on its own by default, but this argument goes through quantitatively—you'll regret not having quantitatively accelerated HIA if you could have.
Very short timelines are pretty intractable. If AGI is actually coming in 5 years by default, unless you think alignment is easy, there just aren't many outs. Of course we still want to try, but intractability does weigh in the prioritization calculation.
Pulling world-saving conditions forward in time is very valuable unless you're very confident extinction will have already happened.
AGI alignment is extremely hard. Therefore solving AGI alignment is not that helpful of an out.
HIA is an out we can play to.
Adult HIA methods aren't fast either, absent big investment
A variant of the argument against reprogenetics goes like this:
Of course, overall, this logic is valid and compelling. If I believed we could do (strong) HIA sooner than a couple decades, I would work on that. If there were actually a Manhattan-Project-scale project for adult human intelligence amplification, I would probably drop what I was working on in reprogenetics and join that effort. I would have some hope of success—assuming that we would have available the scientists, equipment, experimental volunteers, and money that would be needed to run several experimental investigations in parallel.
However, short of a Manhattan Project, I'm somewhat skeptical of adult HIA being a faster bet than reprogenetics. There are several reasons, which I'll list here. But I want to summarize the main reasons more briefly:
The reasons in more detail:
Reprogenetics has good-enough data on intelligence; adult HIA does not.
Reprogenetics would probably work for strong HIA; adult interventions may or may not work well for strong HIA.
There are general technical obstacles to adult HIA.
Reprogenetics has strong momentum, in terms of scientific foundations, user motivation, and user on-ramp.
Therefore, adult HIA is fairly likely to take a rather long time.
Now, all of this having been said, I could easily be wrong. I've focused largely on reprogenetics, because that's what I think will work. I'm enthusiastic about roadmapping and research on adult HIA in general, and on specific methods. Hopefully, those interested in adult HIA would take these as non-dealbreaker constraints, to chew through or work around or creatively break.
Takeaways