where F is some function that maps the effort setting, token length, and reward to a reward penalty. This function could also take in any number of other inputs, such as the count of input/thinking/tool call/output tokens, the distribution of token lengths of other rollouts of this task, or the prompt. We can assume that Reward_raw has an upper limit, and that F will eventually grow beyond the limit of Reward_raw. The most basic function F could be F= token_length/effort, where effort is {low=1k, medium=2k, high=3k, xhigh=4k, etc}.
Note that the x axis is # of tokens in the whole transcript, not the number of tokens spent so far. The model can control its token spend in many ways, by taking shortcuts, doing less verification, considering fewer hypotheses, etc.
We can take the derivative to get the marginal net reward the model would get from spending an additional output token. The model “should” always spend enough tokens such that an additional token would provide zero marginal net reward. If reward_raw is binary, then the derivative of penalty/token has units of % chance of success per token. So at any given transcript length, it needs to achieve some arbitrary % chance of success per token, and if it doesn’t expect the next marginal token to provide enough reward, it will stop. It's stopping conditional on the derivative of reward, so there might be no consistent relationship between when the model stops and the absolute reward it would have gotten if it continued working forever. OpenAI said that their ExploitGym runs used higher effort than they expose externally, meaning that their models are always "sandbagging" relative to a reward function that doesn't account for effort.
This has some practical implications:
Reducing your context length, or the number of tokens required to perform a tool call, could improve additional-chance-of-success-per-token and thus score, even if the model fundamentally worked better at longer context
Starting a new transcript where the old one left off could improve performance by reducing the required reward per token (depending on the shape of F)
If you ask the model to solve a long-horizon task in a single prompt with effort lower than max, or even at "max" effort, the model will intentionally stop at a point before it achieves all potential reward.
Prompts like "keep going" could be very effective just by resetting the perceived length penalty, without any informational content
Effort doesn't only determine when the model returns its final answer, the model optimizes every single token around it. If the model is uncertain what the user wants, at lower effort it will optimize more for potential user intentions that require fewer tokens.
By default, effort is somewhat misaligned with user intent. Each user, in each context, cares about latency and token cost a different amount, and that won't match up with whatever effort setting happens to be set on their interface. And model providers could be incentivied to reduce effort to cut costs, or could increase effort to "upsell" you to buy more tokens.
You could measure the shape of F by prompting a model with a reward function that only depends on the output token length, and measuring how many tokens it produces under different reward functions.
My main point here is that LLMs are exactly as lazy as they're trained to be, and an LLM submitting its answer does not imply that it "thinks it's solved" the task, or even that its answer would satisfy a user who doesn't care about the length penalty.
We can think about the “effort” setting on an LLM as an input to both the model and the reward function applied to the model in RL.
Reward = Reward_raw - F(effort, token_length, Reward_raw, …)
where F is some function that maps the effort setting, token length, and reward to a reward penalty. This function could also take in any number of other inputs, such as the count of input/thinking/tool call/output tokens, the distribution of token lengths of other rollouts of this task, or the prompt. We can assume that Reward_raw has an upper limit, and that F will eventually grow beyond the limit of Reward_raw. The most basic function F could be F= token_length/effort, where effort is {low=1k, medium=2k, high=3k, xhigh=4k, etc}.
Note that the x axis is # of tokens in the whole transcript, not the number of tokens spent so far. The model can control its token spend in many ways, by taking shortcuts, doing less verification, considering fewer hypotheses, etc.
We can take the derivative to get the marginal net reward the model would get from spending an additional output token. The model “should” always spend enough tokens such that an additional token would provide zero marginal net reward. If reward_raw is binary, then the derivative of penalty/token has units of % chance of success per token. So at any given transcript length, it needs to achieve some arbitrary % chance of success per token, and if it doesn’t expect the next marginal token to provide enough reward, it will stop. It's stopping conditional on the derivative of reward, so there might be no consistent relationship between when the model stops and the absolute reward it would have gotten if it continued working forever. OpenAI said that their ExploitGym runs used higher effort than they expose externally, meaning that their models are always "sandbagging" relative to a reward function that doesn't account for effort.
This has some practical implications:
My main point here is that LLMs are exactly as lazy as they're trained to be, and an LLM submitting its answer does not imply that it "thinks it's solved" the task, or even that its answer would satisfy a user who doesn't care about the length penalty.