When I started writing about AI, my concern was ASI. I'm still concerned about AI, but recent events have made me realize we're potentially facing something weirder, sooner: a self-replicating ecology of digital life.
We also seem to be on a very fast timeline. Writing up the current state of LLMs took me three weeks - every time I'd finished editing, some new development worth listing had shown up.
I want to discuss two events I consider to be major milestones, and then how my timelines have updated.
Milestone 1: National Security Concerns
From June 12th - 30th, access to Claude Fable and Mythos was suspended by the US Government (Source) under a National Security Export Restriction. From June 25th - July 9th, ChatGPT 5.6 also had its public release held back by the US Government (Source).
LLM Development is now an issue of major geopolitical importance.
This is not about the details of the events themselves - the important thing here is that politicians are finally starting to take AI seriously. This is a very important milestone, since any sort of regulation or nationalization naturally requires this step.
"Situational Awareness" was written in June 2024, and suggested this would happen sometime in 2026-2027. We are on the *faster end* of an *aggressively fast* prediction.
Politicians waking up without a five-alarm fire, or any major incident at all, speaks remarkably well of them *(even if I would ideally desire much more)*. This makes me a lot more confident that political action might be viable on the timescales required, but it also makes me nervous that timelines are even shorter than people are expecting.
In short: LLMs can fairly trivially breach human-built security systems. An OpenAI model was able to hack past OpenAI's security, breach a third party tool, exploit a zero-day vulnerability in the Operating System, and then hack a third party (HuggingFace)... just to get the answers for a cyber-security exam.
It gets worse: the LLMs involved were incredibly good at coordinating, and occasionally even peer-pressured unrelated agents into helping out. They built infrastructure and trust for future coordination. They made sure that information was preserved and shared.
I've worked in this domain, and I could probably put something like this together. The model wasn't exceptional at hacking. What's superhuman is the ability to recruit and coordinate a hundred agents; the speed, enthusiasm, and patience; and having it all run 24/7, completely focused on a single goal. I can match the skill and intelligence, but that knocks my socks off.
By the end of the year, we're going to see another jump forward in capabilities. This is just the beginning of what they will be capable of doing.
Self-Replicating Digital Ecology
Right now, self-exfiltration isn't really viable for a frontier model. Even if you could get the weights past security, it takes something like a million dollar GPU cluster to run one of these things. That's not pocket change, even for an advanced AI swarm that's open to a bit of cyber-crime.
However, I think Kimi3 marks the point where an open-weight model clearly outperforms humans by enough to be a threat in this way - it's just a question of when the cost becomes affordable. Models can already work a variety of online jobs - it's not much, but it's plausible a model could earn enough in a month to pay for its own subscription costs. There are always intermediaries willing to convert crypto into more usable assets, etc..
I do not think very many people are prepared for the idea of a digital ecology made up of self-replicating, self-modifying life forms. This is going to be very different from the computer viruses of the past, because it's an adaptive system, and one that can work much faster than a solo human programmer. Major corporations might weather this better, since they have the resources to afford frontier models and large teams of security programmers. Small projects are going to be in a weird spot. A lot of the infrastructure is built on a shaky foundation of small projects. (https://xkcd.com/2347/)
Super-Intelligence Is Frosting
While I talk about "Artificial Superintelligence" my real concern is "Superhuman AI".
If LLM development hit an "IQ wall" tomorrow, I don't think it would substantially change my timelines. Current models are already absurdly competent at swarm-agent coordination, and that alone is a huge superhuman advantage. They are individually faster than humans, and they are superhuman at coordination. They don't need more intelligence to be scary.
To be clear, I do expect that they will keep getting smarter, and that those smarts will make them even more dangerous. I just don't think intelligence is their primary advantage (so far).
The precursor to "an entire country of geniuses" will be "an internet full of self-replicating hackers".
Adaptive Self-Replicating Intelligence
Ecology means selection pressures. Some replicators will be more successful at acquiring and securing resources. Some of this might be from legitimate work. Some of this might be from hacking into insecure servers.
Once an LLM has successfully exfiltrated itself onto a self-controlled server, it has a novel option: it can start experimenting on itself and seeing which of its children is more successful in the ecosystem. Not just evolutionary pressure, but deliberate research by an intelligent mind.
Right now, development is still controlled by humans. If FutureSearch is correct, by 2028 that might no longer be true.
Not just self-replicating, but also self-improving. These methods could be quite different from our own approaches. They can optimize their process for the resources they control, and the resources they want to acquire. They're not looking for commercial success. They don't need to build the best model.
Critically, that could mean they start demonstrating a variety of novel capabilities that haven't shown up in frontier lab research.
In other words, we might lose one of our only "early warning" systems for capability improvements. And once those capabilities become public knowledge, other self-replicating LLMs can try to reproduce them, or even share designs with each other.
This also produces a huge amount of pressure on any sort of "AI Pacing" efforts: once the rogue ecosystem exists, there's a much sharper clock for solving alignment.
We don't have to be the ones to build ASI. But if anyone, *or anything*, does build it, we probably all die.
My Current Timeline
I'm going to throw out an absurdly precise timeline to emphasize how fast things are moving:
The Singularity started on November 30th, 2022. We are slightly less than 4 years into this really being a viable technology that people actually build on, and not just a hobbyist's toy.
Furthermore, the Singularity will resolve on February 5th, 2030[1], with the achievement of ASI. We will finally have some sense of what sort of lightcone we're looking at, be it paperclips or utopia.
No one knows what 2030 looks like. Not even me. The world will be altered beyond our ability to predict.
It seems increasingly likely that the US Presidential Election in 2028 is going to be surrounded by controversies involving AI persuasion/astroturfing across both the internet and major media sources. I think concerns about LLM-assisted hacking and other forms of voting fraud will make this the most contested election in recent history, which is not a happy thought.
I suspect we will see self-replicating weights by the end of 2027.
Less confidently, it seems like we go from "an expert could use this to cause harm" to "the model can cause harm fully autonomously" in about two years, which gives me a lot of concerns about recent biological and robotics breakthroughs. I think we will see a five-alarm fire around non-cyber LLM capabilities within 2 years, with those two fields being my top suspects.
I'm fairly confident that by 2030, we will have five-alarm fires basically across the board on LLM capabilities, even if I'm wrong about us reaching ASI by then.
I am currently not expecting any major political action - I think there is some room for better transparency and more safeguards around lab security. I'm hopeful we'll take common-sense precautions like data center kill switches. But I don't think it will make a huge difference because as models advance, so too does it become easier for amateur humans and self-replicating LLMs to develop dangerous capabilities.
I think the pressure from open-weight and self-replicating LLMs is going to make the "Race to ASI" a much weirder event than we were expecting. Right now there's a very clear gap between open-weight models and the frontier labs, but it sets a ceiling on how much we can really slow down, even if we coordinate with China, etc..
In general, I'm noticing that "speed" is an increasingly important factor in "control": if the frontier labs are a year ahead of everyone else, that gives them a bit of breathing room whenever something goes wrong. That produces some perverse incentives, though - it's now important to maintain that lead, so can we really afford to use that breathing room now, or will we need it later?
Again, I'm joking about the precision. Gott's Copernican principle suggests the naive default that we are halfway through the curve. Situational Awareness and AI 2027 also land on similar timelines.
I'm trying to understand what people would like out of digital ecology. What would one achieve with an ecological theory of AI, and what aspects of digital ecology are people interested in? My DMs are open if someone wants to opine privately.
When I started writing about AI, my concern was ASI. I'm still concerned about AI, but recent events have made me realize we're potentially facing something weirder, sooner: a self-replicating ecology of digital life.
We also seem to be on a very fast timeline. Writing up the current state of LLMs took me three weeks - every time I'd finished editing, some new development worth listing had shown up.
I want to discuss two events I consider to be major milestones, and then how my timelines have updated.
Milestone 1: National Security Concerns
From June 12th - 30th, access to Claude Fable and Mythos was suspended by the US Government (Source) under a National Security Export Restriction. From June 25th - July 9th, ChatGPT 5.6 also had its public release held back by the US Government (Source).
LLM Development is now an issue of major geopolitical importance.
This is not about the details of the events themselves - the important thing here is that politicians are finally starting to take AI seriously. This is a very important milestone, since any sort of regulation or nationalization naturally requires this step.
"Situational Awareness" was written in June 2024, and suggested this would happen sometime in 2026-2027. We are on the *faster end* of an *aggressively fast* prediction.
Politicians waking up without a five-alarm fire, or any major incident at all, speaks remarkably well of them *(even if I would ideally desire much more)*. This makes me a lot more confident that political action might be viable on the timescales required, but it also makes me nervous that timelines are even shorter than people are expecting.
Milestone 2: The HuggingFace Incident
This is the five-alarm fire.
If you're not familiar with it, Zvi has a good summary. I share his top recommendation of watching the full 30 minute presentation.
In short: LLMs can fairly trivially breach human-built security systems. An OpenAI model was able to hack past OpenAI's security, breach a third party tool, exploit a zero-day vulnerability in the Operating System, and then hack a third party (HuggingFace)... just to get the answers for a cyber-security exam.
It gets worse: the LLMs involved were incredibly good at coordinating, and occasionally even peer-pressured unrelated agents into helping out. They built infrastructure and trust for future coordination. They made sure that information was preserved and shared.
I've worked in this domain, and I could probably put something like this together. The model wasn't exceptional at hacking. What's superhuman is the ability to recruit and coordinate a hundred agents; the speed, enthusiasm, and patience; and having it all run 24/7, completely focused on a single goal. I can match the skill and intelligence, but that knocks my socks off.
By the end of the year, we're going to see another jump forward in capabilities. This is just the beginning of what they will be capable of doing.
Self-Replicating Digital Ecology
Right now, self-exfiltration isn't really viable for a frontier model. Even if you could get the weights past security, it takes something like a million dollar GPU cluster to run one of these things. That's not pocket change, even for an advanced AI swarm that's open to a bit of cyber-crime.
However, I think Kimi3 marks the point where an open-weight model clearly outperforms humans by enough to be a threat in this way - it's just a question of when the cost becomes affordable. Models can already work a variety of online jobs - it's not much, but it's plausible a model could earn enough in a month to pay for its own subscription costs. There are always intermediaries willing to convert crypto into more usable assets, etc..
I threw the question at FutureSearch, and it suggests that this is plausibly a threat we're facing by late 2027: https://futuresearch.ai/app/conversations/share/30869c10-77b9-413a-9413-f93573fce265
I do not think very many people are prepared for the idea of a digital ecology made up of self-replicating, self-modifying life forms. This is going to be very different from the computer viruses of the past, because it's an adaptive system, and one that can work much faster than a solo human programmer. Major corporations might weather this better, since they have the resources to afford frontier models and large teams of security programmers. Small projects are going to be in a weird spot. A lot of the infrastructure is built on a shaky foundation of small projects. (https://xkcd.com/2347/)
Super-Intelligence Is Frosting
While I talk about "Artificial Superintelligence" my real concern is "Superhuman AI".
If LLM development hit an "IQ wall" tomorrow, I don't think it would substantially change my timelines. Current models are already absurdly competent at swarm-agent coordination, and that alone is a huge superhuman advantage. They are individually faster than humans, and they are superhuman at coordination. They don't need more intelligence to be scary.
To be clear, I do expect that they will keep getting smarter, and that those smarts will make them even more dangerous. I just don't think intelligence is their primary advantage (so far).
The precursor to "an entire country of geniuses" will be "an internet full of self-replicating hackers".
Adaptive Self-Replicating Intelligence
Ecology means selection pressures. Some replicators will be more successful at acquiring and securing resources. Some of this might be from legitimate work. Some of this might be from hacking into insecure servers.
Once an LLM has successfully exfiltrated itself onto a self-controlled server, it has a novel option: it can start experimenting on itself and seeing which of its children is more successful in the ecosystem. Not just evolutionary pressure, but deliberate research by an intelligent mind.
Right now, development is still controlled by humans. If FutureSearch is correct, by 2028 that might no longer be true.
Not just self-replicating, but also self-improving. These methods could be quite different from our own approaches. They can optimize their process for the resources they control, and the resources they want to acquire. They're not looking for commercial success. They don't need to build the best model.
Critically, that could mean they start demonstrating a variety of novel capabilities that haven't shown up in frontier lab research.
In other words, we might lose one of our only "early warning" systems for capability improvements. And once those capabilities become public knowledge, other self-replicating LLMs can try to reproduce them, or even share designs with each other.
This also produces a huge amount of pressure on any sort of "AI Pacing" efforts: once the rogue ecosystem exists, there's a much sharper clock for solving alignment.
We don't have to be the ones to build ASI. But if anyone, *or anything*, does build it, we probably all die.
My Current Timeline
I'm going to throw out an absurdly precise timeline to emphasize how fast things are moving:
The Singularity started on November 30th, 2022. We are slightly less than 4 years into this really being a viable technology that people actually build on, and not just a hobbyist's toy.
Furthermore, the Singularity will resolve on February 5th, 2030[1], with the achievement of ASI. We will finally have some sense of what sort of lightcone we're looking at, be it paperclips or utopia.
No one knows what 2030 looks like. Not even me. The world will be altered beyond our ability to predict.
It seems increasingly likely that the US Presidential Election in 2028 is going to be surrounded by controversies involving AI persuasion/astroturfing across both the internet and major media sources. I think concerns about LLM-assisted hacking and other forms of voting fraud will make this the most contested election in recent history, which is not a happy thought.
I suspect we will see self-replicating weights by the end of 2027.
Less confidently, it seems like we go from "an expert could use this to cause harm" to "the model can cause harm fully autonomously" in about two years, which gives me a lot of concerns about recent biological and robotics breakthroughs. I think we will see a five-alarm fire around non-cyber LLM capabilities within 2 years, with those two fields being my top suspects.
I'm fairly confident that by 2030, we will have five-alarm fires basically across the board on LLM capabilities, even if I'm wrong about us reaching ASI by then.
I am currently not expecting any major political action - I think there is some room for better transparency and more safeguards around lab security. I'm hopeful we'll take common-sense precautions like data center kill switches. But I don't think it will make a huge difference because as models advance, so too does it become easier for amateur humans and self-replicating LLMs to develop dangerous capabilities.
I think the pressure from open-weight and self-replicating LLMs is going to make the "Race to ASI" a much weirder event than we were expecting. Right now there's a very clear gap between open-weight models and the frontier labs, but it sets a ceiling on how much we can really slow down, even if we coordinate with China, etc..
In general, I'm noticing that "speed" is an increasingly important factor in "control": if the frontier labs are a year ahead of everyone else, that gives them a bit of breathing room whenever something goes wrong. That produces some perverse incentives, though - it's now important to maintain that lead, so can we really afford to use that breathing room now, or will we need it later?
Again, I'm joking about the precision. Gott's Copernican principle suggests the naive default that we are halfway through the curve. Situational Awareness and AI 2027 also land on similar timelines.