LLM Intelligence, measured by Starburst, has been basically linear over time since that start of the reasoning era.
By Chapin Lenthall-Cleary and Cole Gaboriault
In summer 2024, my friend Cole and I started a hobby research program into perceptions of (human) intelligence. We started by creating a bespoke 1-5 intelligence scale[1] and rating people we knew. Our plan was to then make a bunch of tests, give them to the people we rated, and see which tests, if any, correlated with our ratings.
One of those tests was Starburst. We’ve talked about it before; in short, it’s a puzzle game where one tries to figure out the laws of physics of a fictional universe. We never did get very many people to play it. But using the results of two early playthroughs (Cole’s and our friend Ryan’s), some further test playthroughs where we pretended to be people of certain intelligences, some a priori reasoning about which parts of the game would be difficult, and a healthy dose of raw intuition, we pieced together a correspondence between Starburst scores and our intelligence scale. Then Ryan suggested we give Starburst to LLMs.
Here's a plot of SOTA[2] model intelligence over time, measured using those Starburst correspondences, along with all four (emphatically not population-representative) human scores for reference (please reach out if you want to play and get us more data):
There's a lot notable here, including the general observations that progress has been rapid, but LLM performance still falls short of the most capable humans we've tested (let alone saturating the benchmark to the level of genius humans or an arbitrary superintelligence).
But the feature of that plot that I find by far most striking is that intelligence over time has been basically linear since the start of the reasoning era. Here, I'll add a line through it:
Okay, wtf?
Obviously, there's some noise and stair-stepping. But the broad trend has been basically linear from the release of gpt-4o (May '24) until now.
What? How? We made a correspondence largely on priors between our single-task benchmark and our bespoke intelligence scale, and throughout the entire reasoning era, even in regimes where I expected walls, the intelligence vs. time trend remained basically linear.
I do not understand how this is happening. My best guess is that the simplest explanation is correct here: our intelligence scale and correspondence approximately capture some deep underlying parameter of intelligence, at least in the ~2-4.5 range, and progress has been linear in this parameter[3]. Even that simplest explanation would be a surprising result with a lot of unanswered questions.
Pretty much as soon as we started evaluating LLMs on Starburst, we also made linear extrapolations of the trend:
The solid orange line shows the date at which we projected models would reach 4.2[4] intelligence, linearly extrapolating the rate from GPT-4o until that point. As early as the release of o3/o4-mini in April 2025, these projections were roughly accurate; since 4.2 wound up being near the bottom of a stair-step, the higher (i.e. later) predictions were more accurate. Over and over again, since ~fall 2024, my reaction has been that the forecasts implied by these extrapolations were insanely fast (~90th percentile of my expectations). Over and over again, they were right.
If we see something similar for 5 intelligence, i.e. that 5 winds up being at the start of a typically-sized stair-step, LLMs would reach genius-level in August or September of 2027; if 5 is at the end of a stairstep, extrapolation suggests it would be reached in roughly April. Obviously there are plenty of other forecasts suggesting and reasons to expect vaguely similar timing, but the agreement from this unreasonably effective is notable. For my part, I'd put my median for reaching genius-level in early 2028, and some of the reason for that delay is the possibility of catastrophe, war, slowdown agreement, etc.
To be clear, reaching genius-level at Starburst doesn't necessarily mean that a model has all abilities of human geniuses. But in the fields where problem-solving ability is the principal limitation, realized model abilities have been in line with what I'd expect from these intelligence values. Models started being able to do advanced undergrad-level physics problems fairly reliably in the mid-to-high 3s. The vague general perception that models were finally good came around the high 3s. The torrent of minor math results started around 4. The Navier-Stokes solution and recent spate of progress on major math, physics, etc. seems to be the product of swarms of models around a 4.5 (obviously unreleased, so treat that as a guess).
Well, fuck.
Appendix: our (and some of our friends’) projections from summer 2025. Whoops.
LLM Intelligence, measured by Starburst, has been basically linear over time since that start of the reasoning era.
By Chapin Lenthall-Cleary and Cole Gaboriault
In summer 2024, my friend Cole and I started a hobby research program into perceptions of (human) intelligence. We started by creating a bespoke 1-5 intelligence scale[1] and rating people we knew. Our plan was to then make a bunch of tests, give them to the people we rated, and see which tests, if any, correlated with our ratings.
One of those tests was Starburst. We’ve talked about it before; in short, it’s a puzzle game where one tries to figure out the laws of physics of a fictional universe. We never did get very many people to play it. But using the results of two early playthroughs (Cole’s and our friend Ryan’s), some further test playthroughs where we pretended to be people of certain intelligences, some a priori reasoning about which parts of the game would be difficult, and a healthy dose of raw intuition, we pieced together a correspondence between Starburst scores and our intelligence scale. Then Ryan suggested we give Starburst to LLMs.
Here's a plot of SOTA[2] model intelligence over time, measured using those Starburst correspondences, along with all four (emphatically not population-representative) human scores for reference (please reach out if you want to play and get us more data):
There's a lot notable here, including the general observations that progress has been rapid, but LLM performance still falls short of the most capable humans we've tested (let alone saturating the benchmark to the level of genius humans or an arbitrary superintelligence).
But the feature of that plot that I find by far most striking is that intelligence over time has been basically linear since the start of the reasoning era. Here, I'll add a line through it:
Okay, wtf?
Obviously, there's some noise and stair-stepping. But the broad trend has been basically linear from the release of gpt-4o (May '24) until now.
What? How? We made a correspondence largely on priors between our single-task benchmark and our bespoke intelligence scale, and throughout the entire reasoning era, even in regimes where I expected walls, the intelligence vs. time trend remained basically linear.
I do not understand how this is happening. My best guess is that the simplest explanation is correct here: our intelligence scale and correspondence approximately capture some deep underlying parameter of intelligence, at least in the ~2-4.5 range, and progress has been linear in this parameter[3]. Even that simplest explanation would be a surprising result with a lot of unanswered questions.
Pretty much as soon as we started evaluating LLMs on Starburst, we also made linear extrapolations of the trend:
The solid orange line shows the date at which we projected models would reach 4.2[4] intelligence, linearly extrapolating the rate from GPT-4o until that point. As early as the release of o3/o4-mini in April 2025, these projections were roughly accurate; since 4.2 wound up being near the bottom of a stair-step, the higher (i.e. later) predictions were more accurate. Over and over again, since ~fall 2024, my reaction has been that the forecasts implied by these extrapolations were insanely fast (~90th percentile of my expectations). Over and over again, they were right.
If we see something similar for 5 intelligence, i.e. that 5 winds up being at the start of a typically-sized stair-step, LLMs would reach genius-level in August or September of 2027; if 5 is at the end of a stairstep, extrapolation suggests it would be reached in roughly April. Obviously there are plenty of other forecasts suggesting and reasons to expect vaguely similar timing, but the agreement from this unreasonably effective is notable. For my part, I'd put my median for reaching genius-level in early 2028, and some of the reason for that delay is the possibility of catastrophe, war, slowdown agreement, etc.
To be clear, reaching genius-level at Starburst doesn't necessarily mean that a model has all abilities of human geniuses. But in the fields where problem-solving ability is the principal limitation, realized model abilities have been in line with what I'd expect from these intelligence values. Models started being able to do advanced undergrad-level physics problems fairly reliably in the mid-to-high 3s. The vague general perception that models were finally good came around the high 3s. The torrent of minor math results started around 4. The Navier-Stokes solution and recent spate of progress on major math, physics, etc. seems to be the product of swarms of models around a 4.5 (obviously unreleased, so treat that as a guess).
Well, fuck.
Appendix: our (and some of our friends’) projections from summer 2025. Whoops.
1: Severely retarded
1.25: Retarded
1.5: Very dumb
1.75: Quite dumb
2: (Ordinary) dumb
2.25: Somewhat dumb
2.5: A bit dumb
2.75: Slightly dumb
3: Neither smart nor dumb
3.25: Slightly smart
3.5: A bit smart
3.75: Somewhat smart
4: (Ordinary) smart
4.25: Quite smart
4.5: Very smart
4.75: Sub-genius
5: Genius
We only include models available at the $20/mo tier, partly to standardize compute budget, but mostly because we are poor.
Or our scale is logarithmic in the underlying real intelligence parameter and progress has been exponential, or whatever other transformation.
Which we set as a criterion for preparing to move to the woods before we started benchmarking in earnest.