I've seen the argument thrown around that time horizon predictions from AECI are greater than those in AI2027 (about ~100-200 hours) therefore we should be worried about imminent RSI[1]. This comes from graphs like this one by Nikola Jurkovic:
This is direct experimental evidence that more accurately reflects the question of researcher uplift in AI2027, so it should be more reliable than the AECI predictions. These measurements may represent either 5.6 Sol coming out in July, or 6 Astra coming out in September, or a mixture of both. You can see a jump in time horizon in the data at ~June which may be where models were switched. I think it's reasonable to compare these because these are frontier-to-frontier internal model comparisons in the same month, although this could represent one lab being behind another.
If we were to eyeball this graph to be like the time horizon estimates above and plot it, we could see whether the predictions and measurements are similar. I took the midpoint of the time horizon bin, and had Claude and I extract data from both graphs for their measures as well:
We can see that the predictions from OpenAI are significantly smaller than the ones expected from AECI. The fitted gap between the two predictions is somewhere between 2-3 doublings, or 2.5-11 months. AECI predictions are weak evidence of time horizons in comparison to direct measures of which we already have access. These are also 50% success rate time horizons, not the 80% time horizons in the AI2027 forecast, which would push the timeline even further into the future.
I think people know that time horizon predictions from AECI are unlikely to be accurate, but it is worth pointing out that we do in fact have some estimates of time horizon for research activity for recent models. I suspect that current statistical evidence is not viable to nowcast uplift, but hindcasts can at least show us which pieces of evidence are more or less reliable.
Half-way between a quick take and a post.
I've seen the argument thrown around that time horizon predictions from AECI are greater than those in AI2027 (about ~100-200 hours) therefore we should be worried about imminent RSI[1]. This comes from graphs like this one by Nikola Jurkovic:
These graphs seem convincing because AECI and time horizon strongly correlate with each other in weaker models.
But we also have an estimate of time horizons relevant to researchers from OpenAI that came out earlier this year:
This is direct experimental evidence that more accurately reflects the question of researcher uplift in AI2027, so it should be more reliable than the AECI predictions. These measurements may represent either 5.6 Sol coming out in July, or 6 Astra coming out in September, or a mixture of both. You can see a jump in time horizon in the data at ~June which may be where models were switched. I think it's reasonable to compare these because these are frontier-to-frontier internal model comparisons in the same month, although this could represent one lab being behind another.
If we were to eyeball this graph to be like the time horizon estimates above and plot it, we could see whether the predictions and measurements are similar. I took the midpoint of the time horizon bin, and had Claude and I extract data from both graphs for their measures as well:
We can see that the predictions from OpenAI are significantly smaller than the ones expected from AECI. The fitted gap between the two predictions is somewhere between 2-3 doublings, or 2.5-11 months. AECI predictions are weak evidence of time horizons in comparison to direct measures of which we already have access. These are also 50% success rate time horizons, not the 80% time horizons in the AI2027 forecast, which would push the timeline even further into the future.
I think people know that time horizon predictions from AECI are unlikely to be accurate, but it is worth pointing out that we do in fact have some estimates of time horizon for research activity for recent models. I suspect that current statistical evidence is not viable to nowcast uplift, but hindcasts can at least show us which pieces of evidence are more or less reliable.
This is related but different to the argument about whether the acceleration of Anthropic internally is 2x. The evidence here should inform that too.
Errors like me eyeballing it, to give one example.