In a research post, Cunningham et al. of METR propose the “‘expenditure horizon’ over an optimization problem as the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget.” Though their work focuses on individual optimization problems (with NanoGPT as the main subject), the expenditure horizon can be meaningful for any task distribution: it’s the budget at which human-only and AI-only work reach the same performance.
Though I like expenditure horizons as an operationalization of AI capability, I have a few critiques which suggest an improved metric, which I’ll call the “X% expenditure share horizon”: at what labor budget is it optimal to spend X% of the budget on agents rather than humans?
In this essay, I’ll explain why this metric is both more useful and likely easier to measure than the expenditure horizon. I think this is an area that’s ripe for concrete follow-up work.
Figure 1. Performance as a function of expenditure on humans or agents. Source: METR.
Pros and Cons of the Expenditure Horizon Metric
Let me begin by saying a bit about what I like about the expenditure horizon metric. First, as we brace for the economic impacts of AI, I think it’s good to begin proposing measures of AI capability that are tied to the human cost of comparable work, and the expenditure horizon takes a shot at this. Another metric of this kind that I am partial to is “autoflation” from theRemote Labor Index: “The percentage decrease in the cost of completing [a] fixed [task] bundle when using the cheapest-possible method to complete each [task] (human [or AI]).”
Second, as Cunningham et al. note, the expenditure horizon implicitly captures how agent performance varies with test-time compute. This is good, since AIs should no longer be understood to have a single “score” on any task. Older metrics like original time horizons don’t have a built-in way to indicate task budgets.
Finally, I like that the expenditure horizon has a meaningful interpretation for a horizon of infinity: this is the point at which agents alone perform better than humans alone at any cost. That’s not quite the same as saying that a marginal labor dollar should always go to agents, though, which brings us to the main con of the expenditure horizon: it doesn’t account for hybrid human-AI labor.
Con 1: AI and Human-Only Performance Is Less Informative Than Hybrid Performance
Cunningham et al. acknowledge that their metric’s exclusion of hybrid work is a limitation: “In some cases we would expect hybrid optimization (humans assisted by AI) to be a significant improvement on autonomous optimization.”
I would state the issue more strongly. For most task distributions where we care to measure AI uplift (especially AI R&D), humans and AIs are complements: work is best done in a hybrid manner. Tasteful human prompting and occasional intervention can augment AI performance, while AIs can cheaply assist with many subtasks as humans work. If both humans and AIs benefit from even small amounts of augmentation by the other, then metrics based on AI and human-only performance (the endpoints in the graph below) will be less informative.
Figure 2. Performance curves are concave for any budget where human-AI complementarity holds. At low budgets, humans don’t have enough time to make progress, so agent-only performance is better than human-only performance. At high budgets, agents eventually get stuck on their own. The expenditure horizon in the context of this figure would be an intermediate budget for which the two endpoints are at the same level, but why care about the endpoints?
To hammer home the intuition for why this is an issue, consider a slight variant of the expenditure horizon that I’ll call the “1-99” expenditure horizon: the task labor budget for which spending 1% on humans and 99% on agents achieves the same performance as spending 99% on humans and 1% on agents.
For cases where between 1-99% of labor costs are spent on humans (most of the task distributions we’d be interested to measure an expenditure horizon for), the 1-99 expenditure horizon is strictly more informative than the original (“0-100”) expenditure horizon. That is, the 1%-human-99%-agent performance tells us more (than agent-only performance) about real-life AI-heavy labor, and the 99%-human-1%-agent performance tells us more (than human-only performance) about real-life human-heavy labor. That the informativeness of the expenditure horizon metric appears lower than slightly perturbed metrics is evidence to me that a better ontology lies in wait.
Con 2: Human-Only Performance Is Inconvenient to Measure
Additionally, human-only performance will inevitably become hard to estimate. For task distributions with significant AI uplift, most new work will be hybrid or AI-only, in part because humans quicklycome to dislike doing AI-uplifted subtasks manually. Especially in the regime we care about (AI R&D), there are only varying levels of hybrid work, making metrics based on human-only performance somewhat noisier. Cunningham et al. also acknowledge this limitation: “Ideally we would compare researcher productivity with and without AI help, but it is very difficult to run an experiment that estimates the effect accurately.”
The seeds of this issue already appear in the NanoGPT analysis. Cunningham et al. study NanoGPT contributions since January 2025, and mostly assume that contributions were human-only for the purpose of their analysis. However, it’s plausible that agents already provide nontrivial labor cost savings; GPT-5.6 Sol conservativelyestimates that 14% of NanoGPT commits from the past 3 months were coauthored by a coding agent. In this case, I believe any corrections for AI-assisted cost savings would be small, but the fact that such corrections are required is a downside of this methodology.
It’s also not even clear why we should care about human-only performance in the first place. Once we’ve grown used to using AI tools in particular ways, that ought to become the new baseline for “human” productivity. To be slightly facetious, it’s not like we particularly care to measure the “human-only” performance of knowledge workers without their laptops!
Let’s now discuss the expenditure share horizon, which I believe addresses these issues.
The Expenditure Share Horizon (ESH)
Thinking about how to direct marginal labor dollars can sharpen our intuitions and help motivate the X% expenditure share horizon. Recall that Cunningham et al.’s expenditure horizon is defined as the task budget for which we would be indifferent between AI-only and human-only labor. But in practice, we never have to choose between AI-only and human-only labor; instead, we allocate marginal labor dollars differently between humans and agents as our budgets grow.
This marginal spending can be expressed by theexpansion path of human and agent spending as the task budget increases. See below for an illustrative diagram. Each parallel red line represents a particular budget that might be allocated to the task(s) we’re measuring. Given a specific budget and AI model, there is a performance-maximizing allocation of human and agent spend. The expansion path for each model intersects each budget line at that optimal share of AI/Human spending. At first, most marginal spending goes towards AIs, since they are fast and cheap, relative to expensive humans who take a while to get their bearings.
Figure 3. Illustrative expansion paths for Human/AI spend for three different models. Important: this is not a graph of model performance relative to expenditure. It’s a graph of optimal spending between human and AI labor, for a smoothly increasing budget. Segments end where human-agent teams “saturate” the task distribution, when the tasks have been ~fully solved.
For very weak models, almost 100% of spending should go towards humans at any budget (green line). For very capable models, a high share of AI spending might be preferable even at higher budgets (orange line). So long as humans are eventually more capable and reliable than a model, the model’s expansion path should be concave. That is, the optimal human share of spending should increase as the budget increases (literally making human labor a luxury good).
This framework suggests a natural way to judge how valuable an agent is for a task distribution, in a way that is sensitive to how human and agent performance scales with expenditure: at what budget is it optimal to spend X% of the budget on inference for the agent (and the rest on human labor)? Call this amount the X% expenditure share horizon, or “ESH” henceforth. The figure below illustrates this definition.
Gray lines are isoquants of performance on the task distribution of interest: sets of (human spend, AI spend) points that all result in the same performance. Dotted lines with slopes 1, and 1/9, respectively, extend from the origin, and intersect the blue expansion path at points where it is optimal to spend 50% and 10% (respectively) of the budget on agents. Projecting those points along their budget lines to the X-axis, we mark the 50% and 10% ESHs for this task distribution.
Isoquant X-axis and Y-axis intercepts correspond to human-only and AI-only performance, respectively. Consistent with our assumption that agents are more cost-effective than humans at low budgets, but lose out at higher budgets, isoquants closer to the origin (lower budgets) have smaller Y-axis intercepts than X-axis intercepts. And isoquants further from the origin (higher budgets) have smaller X-axis intercepts, and larger or nonexistent Y-axis intercepts, consistent with the idea that humans eventually outperform AIs alone.
There is a particular isoquant—marked in purple in the figure—at which the X-axis and Y-axis intercepts (points A and B in the figure) are equidistant from the origin. This is precisely Cunningham et al.’s expenditure horizon.
But the figure above should help illustrate that the isoquant endpoints aren’t very related to real-world hybrid performance. Equal performance can be attained much more cheaply by augmenting humans and agents with each other, which is what happens in practice. And performance at the endpoints is very sensitive to small amounts of augmentation, which gets at the “1-99” expenditure horizon argument from before. Thus, ESHs speak more informatively about hybrid performance, addressing Con 1 from above: they are tied to the optimal way that a given labor budget can be spent between humans and AIs.
Random Nice Properties of ESHs
I’m relatively partial to ESHs. Three things I like about them: they are meaningful for unbounded values; they might have bearing on economy-wide measures of AI adoption; and they appear relatively more tractable to measure than expenditure horizons.
Unbounded ESHs Are Meaningful
I mentioned earlier that I like when unbounded metrics, like the original expenditure horizon, have a meaningful interpretation for a horizon of infinity. ESHs fare well here: an AI has an undefined (or infinite) X% ESH when it is always optimal to spend more than X% of one’s budget on the AI.
Figure 5. Illustrative plot of optimal budget fraction for several AI models as a function of labor budget. GPT-7 has a bounded 90% ESH, but an unbounded 75% ESH, representing the fact that it’s always optimal to spend more than 75% of one’s labor budget on GPT-7 over humans, on this task distribution.
I like that the metric is also able to identify Pareto superhuman (“GPT-∞”) models, which are allocated every marginal dollar. They’re simply models with an unbounded X% ESH for all X!
ESHs and the Macroeconomics of AI
Additionally, ESHs might provide intuitions for how to expect the macroeconomics of AI to play out. Now that AI revenues are within roughly 2 orders of magnitude of US labor spending, it would be great if real-world economically-relevant-task benchmarking could tell us something about the fraction of total labor spending to expect to go towards humans versus tokens.
The ESH can provide fuzzy but nonzero signal here; namely, if X is the largest value in (0,1) for which measured X% ESHs are unbounded across benchmark suites relevant to a particular industry, we might expect X to be a (somewhat tight, if your tasks are good) upper bound for the fraction of spending going towards tokens rather than humans in that industry. I won’t expound on this idea here, and there are a bunch of ways it can fail to hold, but I do think this is a fruitful topic for future work.
ESHs Are Tractable to Measure
Finally, I think a pro of ESHs is that they appear tractable to measure. Namely, we can run RCTs where humans are given a task, and are allocated a random amount of real-world working time, as well as random token budgets. ESHs can be fit to triples of (human spending, token spending, and resulting performance), where we can infer the optimal (or at least best-performing) AI budget share at various budgets.
Measuring ESHs doesn’t suffer from the points I made above about Cunningham et al.’s expenditure horizons: it’s an evaluation of hybrid human-AI work rather than tricky human-only work. Additionally, ESHs might be possible to regress from “offline” labor data, such as traces of human-agent work, to the extent that heterogeneous human wage data can be inferred, and to the extent that humans naturally divide labor between themselves and AIs in a near-optimal way. This is another topic for future work.
That’s not to say measuring or operationalizing ESHs is actively easy. For the sake of simplicity in this essay, I have so far neglected several complicating factors when it comes to discussing “performance on a task distribution.” First, I have said little about how different humans achieve different performances, both on their own, and because they might use agents more or less effectively. Second, we might simply fail to create well-scoped task distributions that both (a) don’t saturate at realistic budgets, and (b) can be undertaken by humans in realistic evaluation timeframes. Finally, I have not considered how humans and AIs interact in the real world—if humans have to wait for AIs, or struggle to do productive work in an overlapping way, or teams of humans work with teams of agents—frictions will certainly appear that make it hard to be sure that any arrangement of AI and human labor spending is “optimal.”
Conclusion
I expect AI labor to become a macroeconomically measurable part of the economy in the next eighteen months, and it seems important to have high-quality methodologies for benchmarking it. Anchoring benchmarking metrics in dollar costs and making them natively responsive to inference spending is the bare minimum; we also have to account for the fact that most AI spending will be part of hybrid work, and thus that marginal dollars will be split differently between AIs and humans at different budgets.
I’ve proposed the expenditure share horizon as a starting point, but my main takeaway from thinking about this problem has been that AI benchmarking and economics are severely underprepared to guide economic policy on AI in the coming landscape.
In a research post, Cunningham et al. of METR propose the “‘expenditure horizon’ over an optimization problem as the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget.” Though their work focuses on individual optimization problems (with NanoGPT as the main subject), the expenditure horizon can be meaningful for any task distribution: it’s the budget at which human-only and AI-only work reach the same performance.
Though I like expenditure horizons as an operationalization of AI capability, I have a few critiques which suggest an improved metric, which I’ll call the “X% expenditure share horizon”: at what labor budget is it optimal to spend X% of the budget on agents rather than humans?
In this essay, I’ll explain why this metric is both more useful and likely easier to measure than the expenditure horizon. I think this is an area that’s ripe for concrete follow-up work.
Figure 1. Performance as a function of expenditure on humans or agents. Source: METR.
Pros and Cons of the Expenditure Horizon Metric
Let me begin by saying a bit about what I like about the expenditure horizon metric. First, as we brace for the economic impacts of AI, I think it’s good to begin proposing measures of AI capability that are tied to the human cost of comparable work, and the expenditure horizon takes a shot at this. Another metric of this kind that I am partial to is “autoflation” from the Remote Labor Index: “The percentage decrease in the cost of completing [a] fixed [task] bundle when using the cheapest-possible method to complete each [task] (human [or AI]).”
Second, as Cunningham et al. note, the expenditure horizon implicitly captures how agent performance varies with test-time compute. This is good, since AIs should no longer be understood to have a single “score” on any task. Older metrics like original time horizons don’t have a built-in way to indicate task budgets.
Finally, I like that the expenditure horizon has a meaningful interpretation for a horizon of infinity: this is the point at which agents alone perform better than humans alone at any cost. That’s not quite the same as saying that a marginal labor dollar should always go to agents, though, which brings us to the main con of the expenditure horizon: it doesn’t account for hybrid human-AI labor.
Con 1: AI and Human-Only Performance Is Less Informative Than Hybrid Performance
Cunningham et al. acknowledge that their metric’s exclusion of hybrid work is a limitation: “In some cases we would expect hybrid optimization (humans assisted by AI) to be a significant improvement on autonomous optimization.”
I would state the issue more strongly. For most task distributions where we care to measure AI uplift (especially AI R&D), humans and AIs are complements: work is best done in a hybrid manner. Tasteful human prompting and occasional intervention can augment AI performance, while AIs can cheaply assist with many subtasks as humans work. If both humans and AIs benefit from even small amounts of augmentation by the other, then metrics based on AI and human-only performance (the endpoints in the graph below) will be less informative.
Figure 2. Performance curves are concave for any budget where human-AI complementarity holds. At low budgets, humans don’t have enough time to make progress, so agent-only performance is better than human-only performance. At high budgets, agents eventually get stuck on their own. The expenditure horizon in the context of this figure would be an intermediate budget for which the two endpoints are at the same level, but why care about the endpoints?
To hammer home the intuition for why this is an issue, consider a slight variant of the expenditure horizon that I’ll call the “1-99” expenditure horizon: the task labor budget for which spending 1% on humans and 99% on agents achieves the same performance as spending 99% on humans and 1% on agents.
For cases where between 1-99% of labor costs are spent on humans (most of the task distributions we’d be interested to measure an expenditure horizon for), the 1-99 expenditure horizon is strictly more informative than the original (“0-100”) expenditure horizon. That is, the 1%-human-99%-agent performance tells us more (than agent-only performance) about real-life AI-heavy labor, and the 99%-human-1%-agent performance tells us more (than human-only performance) about real-life human-heavy labor. That the informativeness of the expenditure horizon metric appears lower than slightly perturbed metrics is evidence to me that a better ontology lies in wait.
Con 2: Human-Only Performance Is Inconvenient to Measure
Additionally, human-only performance will inevitably become hard to estimate. For task distributions with significant AI uplift, most new work will be hybrid or AI-only, in part because humans quickly come to dislike doing AI-uplifted subtasks manually. Especially in the regime we care about (AI R&D), there are only varying levels of hybrid work, making metrics based on human-only performance somewhat noisier. Cunningham et al. also acknowledge this limitation: “Ideally we would compare researcher productivity with and without AI help, but it is very difficult to run an experiment that estimates the effect accurately.”
The seeds of this issue already appear in the NanoGPT analysis. Cunningham et al. study NanoGPT contributions since January 2025, and mostly assume that contributions were human-only for the purpose of their analysis. However, it’s plausible that agents already provide nontrivial labor cost savings; GPT-5.6 Sol conservatively estimates that 14% of NanoGPT commits from the past 3 months were coauthored by a coding agent. In this case, I believe any corrections for AI-assisted cost savings would be small, but the fact that such corrections are required is a downside of this methodology.
It’s also not even clear why we should care about human-only performance in the first place. Once we’ve grown used to using AI tools in particular ways, that ought to become the new baseline for “human” productivity. To be slightly facetious, it’s not like we particularly care to measure the “human-only” performance of knowledge workers without their laptops!
Let’s now discuss the expenditure share horizon, which I believe addresses these issues.
The Expenditure Share Horizon (ESH)
Thinking about how to direct marginal labor dollars can sharpen our intuitions and help motivate the X% expenditure share horizon. Recall that Cunningham et al.’s expenditure horizon is defined as the task budget for which we would be indifferent between AI-only and human-only labor. But in practice, we never have to choose between AI-only and human-only labor; instead, we allocate marginal labor dollars differently between humans and agents as our budgets grow.
This marginal spending can be expressed by the expansion path of human and agent spending as the task budget increases. See below for an illustrative diagram. Each parallel red line represents a particular budget that might be allocated to the task(s) we’re measuring. Given a specific budget and AI model, there is a performance-maximizing allocation of human and agent spend. The expansion path for each model intersects each budget line at that optimal share of AI/Human spending. At first, most marginal spending goes towards AIs, since they are fast and cheap, relative to expensive humans who take a while to get their bearings.
Figure 3. Illustrative expansion paths for Human/AI spend for three different models. Important: this is not a graph of model performance relative to expenditure. It’s a graph of optimal spending between human and AI labor, for a smoothly increasing budget. Segments end where human-agent teams “saturate” the task distribution, when the tasks have been ~fully solved.
For very weak models, almost 100% of spending should go towards humans at any budget (green line). For very capable models, a high share of AI spending might be preferable even at higher budgets (orange line). So long as humans are eventually more capable and reliable than a model, the model’s expansion path should be concave. That is, the optimal human share of spending should increase as the budget increases (literally making human labor a luxury good).
This framework suggests a natural way to judge how valuable an agent is for a task distribution, in a way that is sensitive to how human and agent performance scales with expenditure: at what budget is it optimal to spend X% of the budget on inference for the agent (and the rest on human labor)? Call this amount the X% expenditure share horizon, or “ESH” henceforth. The figure below illustrates this definition.
Figure 4. Illustrative expenditure horizons and expenditure share horizons plotted together.
Gray lines are isoquants of performance on the task distribution of interest: sets of (human spend, AI spend) points that all result in the same performance. Dotted lines with slopes 1, and 1/9, respectively, extend from the origin, and intersect the blue expansion path at points where it is optimal to spend 50% and 10% (respectively) of the budget on agents. Projecting those points along their budget lines to the X-axis, we mark the 50% and 10% ESHs for this task distribution.
Isoquant X-axis and Y-axis intercepts correspond to human-only and AI-only performance, respectively. Consistent with our assumption that agents are more cost-effective than humans at low budgets, but lose out at higher budgets, isoquants closer to the origin (lower budgets) have smaller Y-axis intercepts than X-axis intercepts. And isoquants further from the origin (higher budgets) have smaller X-axis intercepts, and larger or nonexistent Y-axis intercepts, consistent with the idea that humans eventually outperform AIs alone.
There is a particular isoquant—marked in purple in the figure—at which the X-axis and Y-axis intercepts (points A and B in the figure) are equidistant from the origin. This is precisely Cunningham et al.’s expenditure horizon.
But the figure above should help illustrate that the isoquant endpoints aren’t very related to real-world hybrid performance. Equal performance can be attained much more cheaply by augmenting humans and agents with each other, which is what happens in practice. And performance at the endpoints is very sensitive to small amounts of augmentation, which gets at the “1-99” expenditure horizon argument from before. Thus, ESHs speak more informatively about hybrid performance, addressing Con 1 from above: they are tied to the optimal way that a given labor budget can be spent between humans and AIs.
Random Nice Properties of ESHs
I’m relatively partial to ESHs. Three things I like about them: they are meaningful for unbounded values; they might have bearing on economy-wide measures of AI adoption; and they appear relatively more tractable to measure than expenditure horizons.
Unbounded ESHs Are Meaningful
I mentioned earlier that I like when unbounded metrics, like the original expenditure horizon, have a meaningful interpretation for a horizon of infinity. ESHs fare well here: an AI has an undefined (or infinite) X% ESH when it is always optimal to spend more than X% of one’s budget on the AI.
Figure 5. Illustrative plot of optimal budget fraction for several AI models as a function of labor budget. GPT-7 has a bounded 90% ESH, but an unbounded 75% ESH, representing the fact that it’s always optimal to spend more than 75% of one’s labor budget on GPT-7 over humans, on this task distribution.
I like that the metric is also able to identify Pareto superhuman (“GPT-∞”) models, which are allocated every marginal dollar. They’re simply models with an unbounded X% ESH for all X!
ESHs and the Macroeconomics of AI
Additionally, ESHs might provide intuitions for how to expect the macroeconomics of AI to play out. Now that AI revenues are within roughly 2 orders of magnitude of US labor spending, it would be great if real-world economically-relevant-task benchmarking could tell us something about the fraction of total labor spending to expect to go towards humans versus tokens.
The ESH can provide fuzzy but nonzero signal here; namely, if X is the largest value in (0,1) for which measured X% ESHs are unbounded across benchmark suites relevant to a particular industry, we might expect X to be a (somewhat tight, if your tasks are good) upper bound for the fraction of spending going towards tokens rather than humans in that industry. I won’t expound on this idea here, and there are a bunch of ways it can fail to hold, but I do think this is a fruitful topic for future work.
ESHs Are Tractable to Measure
Finally, I think a pro of ESHs is that they appear tractable to measure. Namely, we can run RCTs where humans are given a task, and are allocated a random amount of real-world working time, as well as random token budgets. ESHs can be fit to triples of (human spending, token spending, and resulting performance), where we can infer the optimal (or at least best-performing) AI budget share at various budgets.
Measuring ESHs doesn’t suffer from the points I made above about Cunningham et al.’s expenditure horizons: it’s an evaluation of hybrid human-AI work rather than tricky human-only work. Additionally, ESHs might be possible to regress from “offline” labor data, such as traces of human-agent work, to the extent that heterogeneous human wage data can be inferred, and to the extent that humans naturally divide labor between themselves and AIs in a near-optimal way. This is another topic for future work.
That’s not to say measuring or operationalizing ESHs is actively easy. For the sake of simplicity in this essay, I have so far neglected several complicating factors when it comes to discussing “performance on a task distribution.” First, I have said little about how different humans achieve different performances, both on their own, and because they might use agents more or less effectively. Second, we might simply fail to create well-scoped task distributions that both (a) don’t saturate at realistic budgets, and (b) can be undertaken by humans in realistic evaluation timeframes. Finally, I have not considered how humans and AIs interact in the real world—if humans have to wait for AIs, or struggle to do productive work in an overlapping way, or teams of humans work with teams of agents—frictions will certainly appear that make it hard to be sure that any arrangement of AI and human labor spending is “optimal.”
Conclusion
I expect AI labor to become a macroeconomically measurable part of the economy in the next eighteen months, and it seems important to have high-quality methodologies for benchmarking it. Anchoring benchmarking metrics in dollar costs and making them natively responsive to inference spending is the bare minimum; we also have to account for the fact that most AI spending will be part of hybrid work, and thus that marginal dollars will be split differently between AIs and humans at different budgets.
I’ve proposed the expenditure share horizon as a starting point, but my main takeaway from thinking about this problem has been that AI benchmarking and economics are severely underprepared to guide economic policy on AI in the coming landscape.