Can decision models like Jev plan ahead? If so, how can we quantify their planning ability?
This is the question that BeelineBench seeks to answer. In this article, I'll explain my methodology and share a few interesting results from the benchmark.
Methodology
The core idea is simple. We present a decision model with a problem domain (like Blocks world) and a list of available actions, and ask it to select the next action in order to solve the problem. We repeat until the decision model has reached a solution, or has reached an evaluation cap. If the decision model explores few unnecessary moves before reaching a solution, the intuition is that the decision model is exhibiting an ability to plan ahead while selecting an action.
To compute the beeline metric, we first count the number of nodes along a shortest path found using a breadth-first search. We then divide by the number of nodes explored by the decision model while finding a solution.
score = (shortest path + 1) / nodes explored with the model
A perfect score is 1, meaning that the decision model reached a solution by exploring exactly the minimum number of nodes.
We include baselines against random-choice and a classic heuristic as deciders. N=100 trials are drawn for each problem domain, and 95% confidence intervals are bootstrapped from the scores. We also compute a number of ancillary metrics and diagnostics.
Results - Beeline Scores
In four of seven problem domains, at least one decider model was able to beat the heuristic-based baseline. All included decider models beat random choice on all problems. Rush Hour was by far the most difficult problem for decider models. The decider models seemed to handle Wikispeedia and "Keys and Doors" quite well.
Cost and Efficiency
A few models appear frequently on the frontier, including Mercury Decide and pplx-decider 1.0.
Limitations
We acknowledge that decider models are highly sensitive to how the problem domain is represented textually. In the included problem domains, we attempted to choose a "natural" or "obvious" representation. We felt this was a better choice than attempting to jointly optimize performance across models by experimenting with textual representations, as this sort of ad hoc experimentation ran the risk of invalidating the benchmark. For now, we consider the ability of the model to understand a reasonable textual representation to be 'part of the benchmark'.
GPT-6 Luna had a high refusal rate for some problem domains, contaminating its score. GPT-6 Luna appears to refuse to answer when the next choice is unclear, but this is an informal observation rather than a validated finding.
Jev has a cap of 255 options per decision. In order to respect Jev's cap in an unbiased way across all deciders (including baselines), we randomly drop options until we are under the cap. In future evaluations, we may limit this behavior to Jev and consider it a handicap specific to Jev.
Next Steps
We plan on including additional models and curating our problem domains. Expense is a factor, so we appreciate donations through Github.
The Repo
BeelineBench is a portable, configurable, reproducible, and re-randomizable benchmark. Each problem domain consists of 100 trials. The trials are randomly drawn from each domain, with an optional master seed for re-randomization. 95% confidence intervals are bootstrapped from each model's result for each problem domain.
The bench is designed to plug in new models, new problems, and to gracefully handle restarts. A PyPi package and Docker containers are included.
Introducion
Can decision models like Jev plan ahead? If so, how can we quantify their planning ability?
This is the question that BeelineBench seeks to answer. In this article, I'll explain my methodology and share a few interesting results from the benchmark.
Methodology
The core idea is simple. We present a decision model with a problem domain (like Blocks world) and a list of available actions, and ask it to select the next action in order to solve the problem. We repeat until the decision model has reached a solution, or has reached an evaluation cap. If the decision model explores few unnecessary moves before reaching a solution, the intuition is that the decision model is exhibiting an ability to plan ahead while selecting an action.
To compute the beeline metric, we first count the number of nodes along a shortest path found using a breadth-first search. We then divide by the number of nodes explored by the decision model while finding a solution.
score = (shortest path + 1) / nodes explored with the modelA perfect score is 1, meaning that the decision model reached a solution by exploring exactly the minimum number of nodes.
We include baselines against random-choice and a classic heuristic as deciders. N=100 trials are drawn for each problem domain, and 95% confidence intervals are bootstrapped from the scores. We also compute a number of ancillary metrics and diagnostics.
Results - Beeline Scores
In four of seven problem domains, at least one decider model was able to beat the heuristic-based baseline. All included decider models beat random choice on all problems. Rush Hour was by far the most difficult problem for decider models. The decider models seemed to handle Wikispeedia and "Keys and Doors" quite well.
Cost and Efficiency
A few models appear frequently on the frontier, including
Mercury Decideandpplx-decider 1.0.Limitations
GPT-6 Lunahad a high refusal rate for some problem domains, contaminating its score.GPT-6 Lunaappears to refuse to answer when the next choice is unclear, but this is an informal observation rather than a validated finding.Next Steps
We plan on including additional models and curating our problem domains. Expense is a factor, so we appreciate donations through Github.
The Repo
BeelineBench is a portable, configurable, reproducible, and re-randomizable benchmark. Each problem domain consists of 100 trials. The trials are randomly drawn from each domain, with an optional master seed for re-randomization. 95% confidence intervals are bootstrapped from each model's result for each problem domain.
The bench is designed to plug in new models, new problems, and to gracefully handle restarts. A PyPi package and Docker containers are included.