Both economists and the AI community are highly interested in forecasting the magnitude and speed of AI’s impact on the economy.

But economists and AI technologists speak different languages. Economists talk in aggregates and dynamics, while technologists talk about capabilities and benchmarks and product releases.

Economists and technologists concern themselves with different scales of time and aggregation, too. A new AI release or benchmark result comes out practically every day. Much key economic data (BEA, BLS, etc.) comes out only yearly, and is aggregated at the level of entire industries.

We basically have no data yet on what the aggregate productivity benefit of 2024’s AI advances will be.

And when the data does come out, and observations are made, we will not know which AI releases and capabilities were the cause.

I am an engineer who is currently directly building large-scale AI automations (an “AI engineer”). What I’ve seen over just the last few months is incredible and undoubtedly important economically. I’m not sure how, but I hope to give economists some “handles” by which they can better observe the impact AI is already having, and the impact it will have.

Main idea: If one looks at AI automation projects at higher resolution, one can get highly suggestive evidence connecting investment opportunities and rates of return (i.e., key quantitative drivers of economic growth) to specific AI capabilities released at specific times.

Here is how it goes:

• A specific AI capability is released (e.g., a reasoning model, or a more reliable instruction-following model, or something more exotic like a computer use model).
• A company (such as the one that I work at) searches for a use case, and finds one (often quite lucrative; quick AI advancement produces lots of low-hanging fruit) and we build a proof-of-concept demo.
  • This takes on the order of a few months from when the relevant AI capability is released, but can be much faster.
  • AI capability connection #1: The timing of demo development reflects the connection of specific AI capabilities to specific categories of market opportunities.
• We do some several iterations of development and deploy a system to production.
  • From what I've seen, this has been fast, even in complex domains. A very large customer support use cases took less than a year to go from first-demo to high-volume production.
  • AI capability connection #2: The speed of AI implementation projects reflects the return on investment (e.g., IRR) achievable with the specific AI capabilities that were used for the demo and implementation.
  • (Development of the system becomes harder as low-hanging fruit are plucked, but also becomes easier as advances in models and tooling turn previously painful use cases simple.)

For example, if you knew what happened in our company, you would find a surprising and very important way in which high-METR-time-horizon models like o3 increase the IRR of AI investment. (I don’t think I can say exactly what it is. But I think you should try to find out!)

In summary, detailed examination of AI implementation projects can help estimate:

• The quantity and speed of investment unlocked by specific AI capabilities (e.g., how much additional investment would a 2x increase in METR time horizon unlock?)
• What the most economically relevant AI capabilities and releases even are

A few other points

• My experience is with enterprise automation projects, which will be a large share of AI’s impact. In sectors with larger products and smaller customers, the impact of specific AI capabilities may be measurable in straightforward ways. For example, the fact that Cursor enjoyed high revenue growth after the release of Claude 3.5 Sonnet (and the associated METR time horizon improvement) is evidence as to the importance of METR time horizon for software engineering value.
• Detailed examination of AI projects also reveals which investments are “crowded in” by AI advances. Notably, my company is being essentially forced to make huge investments in conventional, non-AI software as a result of high AI demand. Knowing this, one can notice signs that the AI automation boom will increase the demand for software. This is currently barely visible at all in labor market data, which shows a quite unhappy picture developers!
  • In fact, I will make this prediction: in the next 2 years, demand for software development professionals will sharply increase.