I (AI 2027 co-author) mostly agree (haven't run this by any other authors).
I agree regarding the GPT-5 release being something to pay attention to.
Whenever I put specific metrics in the text it was supposed to refer to the end of the time period (so "mid 2025" would be end of August since that's that's 2/3 through the year), but I never made that clear anywhere, my bad and I think the middle of the period is a natural interpretation as well. You can see that on the side panel on the right the date starts at Aug 2025.
But either way, waiting until the end of Aug probably won't change the directional update.
What these results do provide is evidence that, indeed, the suggested AI 2027 timelines are unlikely to be met; this should perhaps nudge us towards slightly longer timelines.
I agree that these results should nudge us toward slightly longer timelines, I think bolding that AI 2027 timelines are unlikely to be met seems like it's taking too much from these few early data points if you're implying that these play a big factor in that conclusion (though I might agreed with your assessment even before these data points depending on what you mean by unlikely).
As you allude to, the AI 2027 teams' current median timelines are all longer than 2027 to varying extents (roughly, 2028-2033). We're currently working on an update to our teams' timelines forecasts and will share an update on our views soon.
If the consensus is that the original timeline was too aggressive, another idea for community effort is rewriting the original plot with a longer-term horizon, say an "AI 2030" narrative, which will be published here and then analyzed in real-time, month by month.
We're in the early stage of working on something like this. It wouldn't be just the original plot then stretched out, it would be our best guess predictions conditional on longer timelines.
Key points:
That said, I am also skeptical we'll hit 85% by end of August (pass@1 definition) - new Claude is unlikely to ship by then and the fact that neither Opus outperforms Sonnet and Codex barely outperforms O3 suggest the remaining tasks are quite difficult.
It looks like GPT-5 is still only 74.9% on SWE-bench?
Re: the imperfection of benchmarks, there is reason to believe SWE-bench scores have improved due to data contamination rather than pure model improvement (see "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason"). It was released in October 2023, far before most current frontier models' knowledge cutoff dates.
Possibly better alternatives would be SWE-bench-live (https://swe-bench-live.github.io/, based on Github tasks) or Live code bench (https://livecodebench.github.io/leaderboard.html, based on LeetCode/AtCoder/Codeforces problems). Interestingly enough, these two benchmarks have wildly different results for the top contenders, 19.26% v/s 80.2%. This seems to be mostly due to only the latter using reasoning models, but not only: "real" in context SWE tasks seem to be harder to figure out than code problems.
I don't have evidence for this, but to me it does seem that AI companies are aware of this and keep using SWE-bench to keep their marketing hype. Data contamination is a known issue and it would be a glaring oversight to think it would not happen with such an "old" and public benchmark.
SWEBench Verified - Anthropic claims 80.2% with parallel test time compute, I think we should count it?
How 'lumpy' is progress on these benchmarks? Historically are we seeing relatively smooth increases if we zoom out, or is the chart flat until there's a sudden spike up?
swe-bench pass @ 1 on Claude sonnet versions has been 33.4% (June - 3.5) -> 49.0% (October) -> 62.3% (Feb - 3.7) -> 72.7% (May -> 4). That's practically linear at 3.5% gain/month. That would extrapolate to end of August at 83%.
With the leaderboard at ~75.2% on July 1, such an extrapolation also gets us to around 82%.
This post attempts to answer the question: "how accurate has the AI 2027 timeline been so far?"
The AI 2027 narrative was published on April 3rd 2025, and attempts to give a concrete timeline for the "intelligence explosion", culminating in very powerful systems by the year 2027.
Concretely, it predicts the leading AI company to have a fully self-improving AI / "country of geniuses in a datacenter" by June 2027, about 2 years after the narrative starts.
Today is mid-July 2025, about 3.5 months after the narrative was posted. This means that we have passed about 13% of the timeline up to the claimed "geniuses in a datacenter" moment. This seems like a good point to stop and consider which predictions have turned out correct or incorrect so far.
Specifically, we should consider the points made in the narrative's first section, "Mid 2025: Stumbling Agents". The term "mid-2025" is fairly ambiguous, but we're definitely at this point in time.[1]
There is a prediction tracker for the narrative, posted in May, that attempts to answer this question. So far it claims "100% accuracy"; however, the resolved bulletpoints from the narrative are either obvious ("The better AI agents are expensive"), very general, or were already true when the narrative was published (for pre-existing agents like Devin and Operator).
A better way to check the narrative's accuracy, in my opinion, is to look for the concrete / quantitative predictions. The chapter offers only two such predictions, both concerning the state-of-the-art (SOTA) scores for two benchmarks: OSWorld and SWEBench.
AI 2027's footnote 9 states:
Specifically, we forecast that they [mid-2025 agents] score 65% on the OSWorld benchmark of basic computer tasks (compared to 38% for Operator and 70% for a typical skilled non-expert human).
OSWorld is a benchmark measuring "Open-Ended Tasks in Real Computer Environments" (like accessing folders and updating spreadsheets).
As mentioned, the SOTA in time of publishing was 38.1% for Operator (using ChatGPT 4o as a base model). Since then there weren't any major improvements; the SOTA is now 45.2%, a 7-point improvement, still lagging 20 points behind the prediction.
An important caveat is that OpenAI has recently announced a new agent model, ChatGPT Agent, with improved capabilities compared to Operator.
OpenAI has mentioned several benchmark improvements for this model in their blog post, however it did not mention OSWorld performance at all.
Given the limited info, my best guess is:
My reasoning: a 20-point jump in capabilities will likely be noticed by the company, who (like most AI companies) tend to post a curated list of the most impressive benchmarks in each of their new models, and so far this hasn't been mentioned.
This post will be updated with further information on OSWorld if it arises.
Footnote 10 states:
We forecast that mid-2025 agents will score 85% on SWEBench-Verified.
SWEBench measures models' capabilities in solving real-world Github issues; SWEBench-Verified is a subset of tasks that is curated by OpenAI for more reliability.
The SOTA in time of publishing was around 65% (achieved by Augment Agent v0 on March 2025). The SOTA today is 75%, a 10-point improvement. We're still 10 points short of the predicted milestone.
It is possible that improvements on the benchmark have been achieved and not yet published, but that seems unlikely.[2]
This post considers near-term predictions, and the details written here are likely to be different a few months or even weeks from now.
Because this analysis focuses on the near term, real-world outcomes are likely to be influenced by minute details; a one-month delay in releasing a new model (say, due to marketing reasons) could completely change the narrative.
Thus, one shouldn't take these fine-grained details too seriously. The main point of AI 2027 is not to give a completely accurate reading of the timelines, but to play out a hypothetical example of what an intelligence explosion might look like. The authors themselves admit that the current narrative is too aggressive and the real timeline will likely be slower.
What these results do provide is evidence that, indeed, the suggested AI 2027 timelines are unlikely to be met; this should perhaps nudge us towards slightly longer timelines.
The first set of predictions (the two agent benchmarks) might take twice as long in reality - say, 6 months to hit those milestones instead of 3.
If we stretch the entire timeline by that factor, everything would run about 2x slower, so a self-improving AI is achieved by 2029 (four years from now) instead of 2027.
This is all getting highly speculative, and can be taken in different directions:
Also, it's clear that benchmarks are not a perfect way to measure capabilities, and often benchmark scores don't reflect the models' real-world utility. I believe benchmarks are often played too optimistically (companies achieve impressive-looking scores by optimizing narrow tasks while real-world performance is disappointing), but the reverse is also possible (a benchmark is unnecessarily difficult and a mediocre score hides impressive real-world performance).
Thus, "SWEBench progress is slower than expected" does not necessarily amount to "agent progress is slower than expected". However, I do believe there's a non-zero correlation between the two statements. And while imperfect, benchmarks are still our best way to test progress, rather than ambiguous discussion on which models "seem better". METR has done impressive research in formalizing benchmarks and gaining insights from partial numerical data.
I believe that frequent discussion on how the timelines play out is valuable; a monthly community post analyzing the AI 2027 timelines can be a great way to check our assumptions.
If the consensus is that the original timeline was too aggressive, another idea for community effort is rewriting the original plot with a longer-term horizon, say an "AI 2030" narrative, which will be published here and then analyzed in real-time, month by month.
Arguably there's still time. In 2.5 months we'll enter Q4 of 2025; at that point, the Stumbling Agents predictions should have been fully played out, and the next chapter of the narrative should start ("Late 2025: The World’s Most Expensive AI").
The new OpenAI agent model does mention SWEBench Verified in its system card, where it is roughly comparable (or slightly behind) o3's performance.