by Hans Gundlach, Jayson Lynch, Matthias Mertens, Neil Thompson

(All researchers are part of MIT Futuretech)

Epistemic Status: This is a work in progress, and we would appreciate criticism as well as suggestions in all aspects of this work.

TLDR: We estimate that general inference efficiency increases 3-10x per year. This rate reflects the decrease in the price of inference after attempting to isolate out market competition and hardware effects. However, there is still significant uncertainty in our estimates.

Intro: Algorithmic Progress

Recent advances in AI have shown that improvements in algorithms can yield performance gains far beyond what hardware improvements alone would provide. For example, innovations in transformer architectures, optimization techniques, and inference-time strategies have contributed to substantial reductions in compute requirements for a given level of capability.

While the rate of algorithmic progress in AI training has been estimated at around 3× efficiency gains per year—effectively tripling the usable training compute annually—there has been little systematic work quantifying progress in inference efficiency. Yet inference is crucial: it determines how cheaply models can be deployed, how many “virtual workers” can operate in parallel, and how accessible powerful AI systems become over time. Progress in inference is also a crucial parameter in models of AI recursive self-improvement (see June-July AI 2027) and future scaling of AI labor. This question has gained urgency in light of advances like RL-based training and inference time scaling, where inference computation can dramatically improve capabilities.

It is difficult to quantify the importance of the sources of inference efficiency improvements. Companies do not generally release inference efficiency statistics. Further, many inference advances involve multiple GPUs or large querying and batching (e.g., speculative sampling, continuous batching, paged attention, etc), which makes it hard to measure hardware usage explicitly. However, if markets for open LLM services are competitive, then changes in the price of inference should be a good proxy for changes in inference efficiency. Therefore, in this analysis, we aim to estimate the historical rate of algorithmic progress in inference efficiency using inference prices, attempting to distinguish ...

Posts

Wikitag Contributions

Comments