by Hans Gundlach, Jayson Lynch, Matthias Mertens, Neil Thompson
(All researchers are part of MIT Futuretech)
Epistemic Status: This is a work in progress, and we would appreciate criticism as well as suggestions in all aspects of this work.
TLDR: We estimate that general inference efficiency increases 3-10x per year. This rate reflects the decrease in the price of inference after attempting to isolate out market competition and hardware effects. However, there is still significant uncertainty in our estimates.
Intro: Algorithmic Progress
Recent advances in AI have shown that improvements in algorithms can yield performance gains far beyond what hardware improvements alone would provide. For example, innovations in transformer architectures, optimization techniques, and inference-time strategies have contributed to... (read 2024 more words →)