We evaluate discrete planning in a single forward pass and find that progress has lagged behind standard capability benchmarks, until the release of GPT-6 Astra. These experiments were conducted with access to a version that does not use reasoning, which is not part of Astra’s public release.
AI systems have made substantial capability improvements over the past year, with many of the largest gains coming from reasoning models that output intermediate steps, known as a chain-of-thought (CoT), before giving a final answer. These reasoning traces are also a useful signal for safety tools, allowing monitors to inspect model reasoning for suspicious intent, reward hacking or other signs of misbehavior.
That opportunity depends on decision-relevant computation being externalized. The case for CoT monitoring is that sufficiently difficult serial computation forces models to use the reasoning trace as working memory, but there is no guarantee this holds as models scale, training methods evolve or new architectures emerge.
To track this, we built an evaluation suite that forces models to respond in a single forward pass, spanning constrained graph search, grid navigation, and finite-horizon planning problems. We find substantial progress in no-CoT planning, particularly with GPT-6 Astra, where accuracy jumped to 41.7% compared to 12.8% for GPT-5.6 Sol.
Planning is a useful setting because much of the computation is serial: a model has to compare paths and account for future consequences. Planning is also central to the threat models CoT monitoring is meant to address. Subverting control measures, sustaining deception under oversight, or executing a multi-step takeover all require an agent to hold and follow a plan while evading monitoring.
Each task varies the number of stages a solution must chain together, such as decision periods in a graph problem or connected room segments in a maze, which we call a planning horizon. Each task is extended to deeper horizons while keeping its structure, so a model must carry constraints correctly through more sequential choices to stay optimal. By increasing the horizon, we can test how far models get before reliability breaks down. Every task requires exactly one integer token answer in an effort to measure whether the required computation can be compressed into a single forward pass.
The planning tasks
The no-CoT planning benchmark contains 600 tasks across three datasets, with horizons from 2 through 8 resulting in a total of 4,200 datapoints. Each task has an optimal answer that is always a single integer token. The datasets include:
Dungeon. A text-only planning puzzle with rooms, doors, switches, keys, and relics. The model must find the minimum cost of reaching the exit while satisfying a variety of constraints.
Gridworld. A symbolic navigation problem on a two-dimensional grid with walls, keys, doors, and a goal. The model must return the minimum path length.
Planning Graph. A decision problem over a small state graph. The model is given transition and reward tables and must return the maximum total reward over the specified horizon.
We extend the same underlying instance as the horizon increases rather than sampling an unrelated problem at each depth. Later horizons add rooms, grid segments, or decision periods while preserving earlier structure. Horizon is therefore a controlled proxy for problem depth, not an exact count of intermediate steps. We also verify that tasks cannot be solved by greedy heuristics and that the minimum path length increases with horizon.
The tasks are synthetic and programmatically generated, which makes training data contamination less likely, lets us generate arbitrary horizons, and gives us ground-truth optimal answers. The tradeoff is that the tasks are less naturalistic than an interactive planning environment. Taken together, these probe the same core ability from different angles: can a model maintain and act on a longer chain of constraints as the problem gets deeper?
Figure 1. Example tasks at different horizons. Each task requires one integer answer and does not allow an explanation or scratchpad. This has been simplified to show only a subset of edges between rooms.
Measuring latent reasoning
A straightforward way to measure no-CoT reasoning capabilities is to evaluate models on standard benchmarks while disabling reasoning. Across 10 OpenAI model releases, from GPT-3.5 to Astra, the average performance on MMLU-Pro, LiveBench, and GPQA Diamond increased by 61.2 percentage points. But these gains do not tell us how much sequential reasoning those answers require, and could come from shallow reasoning or training data contamination. One way to understand more clearly what kind of latent reasoning is possible is to look at planning tasks where difficulty corresponds to depth.
In no-CoT planning we see models improving on shallow reasoning tasks over time, but making little to no progress on deep reasoning. Astra marks a substantial improvement in latent planning. Accuracy across the full benchmark rises from 12.8% for GPT-5.6 Sol to 41.7%, with gains extending beyond the shallowest horizon, although performance still deteriorates as planning depth increases. A UK AISI evaluation found that the model’s no-CoT mathematics time horizon at 50% reliability increased from 3.6 minutes for GPT-5.6 Sol to 30.9 minutes for Astra. This measures task-completion difficulty rather than the sequential planning depth measured here and AISI cautioned contamination could inflate the estimate. These experiments were conducted with access to a version that does not use reasoning, which is not part of Astra’s public release.
Figure 2. Standard benchmark performance compared to no-CoT planning benchmark across model releases. Standard capability is the equal-weighted mean of MMLU-Pro, LiveBench, and GPQA Diamond.
Astra shifts the boundary of latent planning
Across the full benchmark, accuracy generally falls as planning horizon increases. The trend is not perfectly monotonic, but models that can sometimes solve shallow horizons become less reliable as the required computation becomes more sequential and combinatorial, and this trend holds for Astra.
Figure 3. Accuracy by planning horizon for GPT-5.5, GPT-5.6 and GPT-6 Astra.
These tasks are substantially easier when models can use reasoning. GPT-5.6 Sol with high reasoning effort achieves 93.3% accuracy across all horizons. However, the tasks remain challenging for GPT-5.6 Luna, particularly at deeper horizons. Sweeping reasoning effort shows that deeper tasks generally elicit more tokens yet achieve lower accuracy at a fixed effort, while additional reasoning improves performance.
Figure 4. GPT-5.6 Luna accuracy versus output tokens, by planning horizon.
To test whether models consider later costs when choosing their first action, we built a control set of matched problem pairs. In each pair, the cheapest first action is optimal in one version but leads to a worse overall outcome in the other. For these problems, models output only their first action. Older models usually chose the cheapest immediate action, while newer models were more likely to account for later costs.
Searching for an upper bound on latent capabilities
The results above measure performance in a single forward pass and represent a lower bound for latent planning. In the wild, models may leverage additional computation power in other ways, such as processing longer input prompts or outputting CoT unrelated to the task.
To test how longer inputs affect planning, we repeated the input prompt up to 50 times and added as many as 1,000 filler tokens (in this case a period surrounded by spaces). For Sol and earlier models, this resulted in only modest gains limited to shallower horizons. But for Astra, we find significant increased performance at all horizons, even with a relatively small number of additional input tokens.
Figure 5. No-CoT planning performance under baseline prompts, 50 repetitions of the full prompt, and 1,000 added filler tokens.
Models can learn latent planning
The results above measure how much models can do latent planning when asked, but this is not a hard limit on what’s possible. The underlying operations (shortest-path search, constrained navigation, and finite-horizon optimization) are familiar, but models have not been trained on these types of tasks directly.
It seems plausible that during training latent reasoning could get reinforced and to test this we ran a small task-specific reinforcement learning experiment on Kimi K2.6 with reasoning disabled. We trained a rank-16 LoRA adapter for 120 updates on 6,048 examples across the three task types and evaluated the final checkpoint on a separate 756-example test set.
Figure 6. Accuracy by planning horizon for the zero-shot Kimi K2.6 baseline and the RL checkpoint on held-out tasks. Error bars show variation across eight trials.
The final checkpoint reached 19.3% exact-match accuracy, 100% strict-integer parse rate. The zero-shot baseline reached 5.2% exact-match accuracy with a 42.2% parse rate; many failures came from the model attempting to write out reasoning despite the single token instruction. Performance still falls with horizon, but the gap shows that models can learn substantially more no-CoT planning when the task is trained directly. We should therefore be mindful of ways training might be reinforcing latent reasoning since this could lead to unexpected jumps in latent planning ability.
Related work
Prior planning and graph benchmarks—including PlanBench, AQA-Bench, GraphArena, NPHardEval, PuzzleBench, and SATBench—typically allow models to externalize computation and likewise find that reliability falls with problem size or constraint complexity. We ask the narrower question of how performance changes with controlled planning depth when the model cannot emit a plan, write code, interact with an environment, or expose chain-of-thought.
A complementary line of work measures no-CoT capability differently. Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models evaluates 14 frontier models on 43 benchmarks and estimates the difficulty of tasks models can complete with 50% reliability using both human completion time and the reasoning-token effort of an anchor model. It finds that broad no-CoT task-completion horizons have grown substantially, doubling roughly annually, although about twice as slowly as time horizons for agentic tasks with CoT. Our benchmark isolates a narrower quantity that aggregate task-completion horizons do not directly measure: reliable serial planning depth. We extend task instances across explicit planning horizons and compare that trajectory with progress on standard capability benchmarks.
Lessons from Studying Two-Hop Latent Reasoning shows how it can be difficult to distinguish genuine hidden composition from memorization and shortcuts. Work on filler-token computation and prompt repetition shows that models can sometimes use non-semantic or repeated tokens to support additional computation, but that these effects depend strongly on training and evaluation details. For monitorability, When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors distinguishes CoT used as post-hoc rationalization from CoT used as computation. Our benchmark is complementary, tracking how the boundary at which visible reasoning becomes load-bearing changes across model releases.
Limitations
Ultimately, we want to understand whether models can carry out seriously harmful actions without being detected by monitors. Planning without CoT is one capability that could contribute to this: a model could compare strategies and anticipate consequences without exposing that reasoning to a monitor. We track this capability as a potential early warning signal.
But models may improve at different kinds of latent reasoning at different rates, and exploiting an immediate opportunity to cause harm may require different capabilities from solving our planning tasks. In a no-CoT evaluation on a subset of BashArena, models completed 61.6% of harmful side tasks while completing the legitimate cover task in only 0.29% of attack trajectories. These results show that some harmful tasks are already achievable without CoT, but do not establish how much internal planning they required. Models were explicitly prompted to attack, which also limits what we can infer about their behavior under ordinary instructions.
We therefore should not treat weak performance on this benchmark as evidence that a model cannot cause serious harm. We nevertheless expect planning to matter for sustained attempts to circumvent safeguards, where a model may need to anticipate later checks, coordinate several actions, and account for how those actions affect its chances of detection. That makes no-CoT planning worth tracking alongside direct evaluations of harmful behavior and monitor evasion, while recognizing that the relationship between these capabilities remains an open question.
Discussion
No-CoT planning has improved substantially, particularly with Astra, but performance still deteriorates as planning horizon increases. Prompt repetition and filler tokens improve Astra’s results even at deeper horizons, showing that baseline scores understate what the model can achieve without writing out intermediate reasoning. These findings leave an important distinction between demonstrating some capacity for latent planning and performing it reliably. The RL result suggests that it is at least partly trainable. Direct optimization improved performance, so future gains may depend less on broad capability scaling than on training methods that encourage computation to remain latent.
One possible explanation for this divergence is that latent planning requires models to maintain and update intermediate state internally rather than in a visible trace. Errors may therefore compound across steps. A model may represent the early stages of a plan correctly but lose track of constraints, costs, or state information as the horizon grows. This remains a hypothesis, however; the benchmark does not distinguish failures of state tracking from failures of search or representation.
No-CoT planning should therefore be treated as a moving capability boundary rather than a fixed limitation of AI systems. The present benchmark offers one way to track that boundary, but its longer-term value will depend on whether it can distinguish generalizable improvements in latent planning from environment-specific adaptation. Identifying which training and architectural changes move the boundary, whether those gains transfer to unfamiliar tasks and longer horizons, and whether they reduce the practical coverage of chain-of-thought-based oversight are promising directions for future work.
The author would like to thank Olivia Watkins, Miles Wang, Yong Zheng-Xin, Jason Wolfe, Micah Carroll, Bowen Baker, Zora Che and Leon Maksin for valuable feedback on drafts of this post.
We evaluate discrete planning in a single forward pass and find that progress has lagged behind standard capability benchmarks, until the release of GPT-6 Astra. These experiments were conducted with access to a version that does not use reasoning, which is not part of Astra’s public release.
AI systems have made substantial capability improvements over the past year, with many of the largest gains coming from reasoning models that output intermediate steps, known as a chain-of-thought (CoT), before giving a final answer. These reasoning traces are also a useful signal for safety tools, allowing monitors to inspect model reasoning for suspicious intent, reward hacking or other signs of misbehavior.
That opportunity depends on decision-relevant computation being externalized. The case for CoT monitoring is that sufficiently difficult serial computation forces models to use the reasoning trace as working memory, but there is no guarantee this holds as models scale, training methods evolve or new architectures emerge.
To track this, we built an evaluation suite that forces models to respond in a single forward pass, spanning constrained graph search, grid navigation, and finite-horizon planning problems. We find substantial progress in no-CoT planning, particularly with GPT-6 Astra, where accuracy jumped to 41.7% compared to 12.8% for GPT-5.6 Sol.
Planning is a useful setting because much of the computation is serial: a model has to compare paths and account for future consequences. Planning is also central to the threat models CoT monitoring is meant to address. Subverting control measures, sustaining deception under oversight, or executing a multi-step takeover all require an agent to hold and follow a plan while evading monitoring.
Each task varies the number of stages a solution must chain together, such as decision periods in a graph problem or connected room segments in a maze, which we call a planning horizon. Each task is extended to deeper horizons while keeping its structure, so a model must carry constraints correctly through more sequential choices to stay optimal. By increasing the horizon, we can test how far models get before reliability breaks down. Every task requires exactly one integer token answer in an effort to measure whether the required computation can be compressed into a single forward pass.
The planning tasks
The no-CoT planning benchmark contains 600 tasks across three datasets, with horizons from 2 through 8 resulting in a total of 4,200 datapoints. Each task has an optimal answer that is always a single integer token. The datasets include:
We extend the same underlying instance as the horizon increases rather than sampling an unrelated problem at each depth. Later horizons add rooms, grid segments, or decision periods while preserving earlier structure. Horizon is therefore a controlled proxy for problem depth, not an exact count of intermediate steps. We also verify that tasks cannot be solved by greedy heuristics and that the minimum path length increases with horizon.
The tasks are synthetic and programmatically generated, which makes training data contamination less likely, lets us generate arbitrary horizons, and gives us ground-truth optimal answers. The tradeoff is that the tasks are less naturalistic than an interactive planning environment. Taken together, these probe the same core ability from different angles: can a model maintain and act on a longer chain of constraints as the problem gets deeper?
Figure 1. Example tasks at different horizons. Each task requires one integer answer and does not allow an explanation or scratchpad. This has been simplified to show only a subset of edges between rooms.
Measuring latent reasoning
A straightforward way to measure no-CoT reasoning capabilities is to evaluate models on standard benchmarks while disabling reasoning. Across 10 OpenAI model releases, from GPT-3.5 to Astra, the average performance on MMLU-Pro, LiveBench, and GPQA Diamond increased by 61.2 percentage points. But these gains do not tell us how much sequential reasoning those answers require, and could come from shallow reasoning or training data contamination. One way to understand more clearly what kind of latent reasoning is possible is to look at planning tasks where difficulty corresponds to depth.
In no-CoT planning we see models improving on shallow reasoning tasks over time, but making little to no progress on deep reasoning. Astra marks a substantial improvement in latent planning. Accuracy across the full benchmark rises from 12.8% for GPT-5.6 Sol to 41.7%, with gains extending beyond the shallowest horizon, although performance still deteriorates as planning depth increases. A UK AISI evaluation found that the model’s no-CoT mathematics time horizon at 50% reliability increased from 3.6 minutes for GPT-5.6 Sol to 30.9 minutes for Astra. This measures task-completion difficulty rather than the sequential planning depth measured here and AISI cautioned contamination could inflate the estimate. These experiments were conducted with access to a version that does not use reasoning, which is not part of Astra’s public release.
Figure 2. Standard benchmark performance compared to no-CoT planning benchmark across model releases. Standard capability is the equal-weighted mean of MMLU-Pro, LiveBench, and GPQA Diamond.
Astra shifts the boundary of latent planning
Across the full benchmark, accuracy generally falls as planning horizon increases. The trend is not perfectly monotonic, but models that can sometimes solve shallow horizons become less reliable as the required computation becomes more sequential and combinatorial, and this trend holds for Astra.
Figure 3. Accuracy by planning horizon for GPT-5.5, GPT-5.6 and GPT-6 Astra.
These tasks are substantially easier when models can use reasoning. GPT-5.6 Sol with high reasoning effort achieves 93.3% accuracy across all horizons. However, the tasks remain challenging for GPT-5.6 Luna, particularly at deeper horizons. Sweeping reasoning effort shows that deeper tasks generally elicit more tokens yet achieve lower accuracy at a fixed effort, while additional reasoning improves performance.
Figure 4. GPT-5.6 Luna accuracy versus output tokens, by planning horizon.
To test whether models consider later costs when choosing their first action, we built a control set of matched problem pairs. In each pair, the cheapest first action is optimal in one version but leads to a worse overall outcome in the other. For these problems, models output only their first action. Older models usually chose the cheapest immediate action, while newer models were more likely to account for later costs.
Searching for an upper bound on latent capabilities
The results above measure performance in a single forward pass and represent a lower bound for latent planning. In the wild, models may leverage additional computation power in other ways, such as processing longer input prompts or outputting CoT unrelated to the task.
To test how longer inputs affect planning, we repeated the input prompt up to 50 times and added as many as 1,000 filler tokens (in this case a period surrounded by spaces). For Sol and earlier models, this resulted in only modest gains limited to shallower horizons. But for Astra, we find significant increased performance at all horizons, even with a relatively small number of additional input tokens.
Figure 5. No-CoT planning performance under baseline prompts, 50 repetitions of the full prompt, and 1,000 added filler tokens.
Models can learn latent planning
The results above measure how much models can do latent planning when asked, but this is not a hard limit on what’s possible. The underlying operations (shortest-path search, constrained navigation, and finite-horizon optimization) are familiar, but models have not been trained on these types of tasks directly.
It seems plausible that during training latent reasoning could get reinforced and to test this we ran a small task-specific reinforcement learning experiment on Kimi K2.6 with reasoning disabled. We trained a rank-16 LoRA adapter for 120 updates on 6,048 examples across the three task types and evaluated the final checkpoint on a separate 756-example test set.
Figure 6. Accuracy by planning horizon for the zero-shot Kimi K2.6 baseline and the RL checkpoint on held-out tasks. Error bars show variation across eight trials.
The final checkpoint reached 19.3% exact-match accuracy, 100% strict-integer parse rate. The zero-shot baseline reached 5.2% exact-match accuracy with a 42.2% parse rate; many failures came from the model attempting to write out reasoning despite the single token instruction. Performance still falls with horizon, but the gap shows that models can learn substantially more no-CoT planning when the task is trained directly. We should therefore be mindful of ways training might be reinforcing latent reasoning since this could lead to unexpected jumps in latent planning ability.
Related work
Prior planning and graph benchmarks—including PlanBench, AQA-Bench, GraphArena, NPHardEval, PuzzleBench, and SATBench—typically allow models to externalize computation and likewise find that reliability falls with problem size or constraint complexity. We ask the narrower question of how performance changes with controlled planning depth when the model cannot emit a plan, write code, interact with an environment, or expose chain-of-thought.
A complementary line of work measures no-CoT capability differently. Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models evaluates 14 frontier models on 43 benchmarks and estimates the difficulty of tasks models can complete with 50% reliability using both human completion time and the reasoning-token effort of an anchor model. It finds that broad no-CoT task-completion horizons have grown substantially, doubling roughly annually, although about twice as slowly as time horizons for agentic tasks with CoT. Our benchmark isolates a narrower quantity that aggregate task-completion horizons do not directly measure: reliable serial planning depth. We extend task instances across explicit planning horizons and compare that trajectory with progress on standard capability benchmarks.
Lessons from Studying Two-Hop Latent Reasoning shows how it can be difficult to distinguish genuine hidden composition from memorization and shortcuts. Work on filler-token computation and prompt repetition shows that models can sometimes use non-semantic or repeated tokens to support additional computation, but that these effects depend strongly on training and evaluation details. For monitorability, When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors distinguishes CoT used as post-hoc rationalization from CoT used as computation. Our benchmark is complementary, tracking how the boundary at which visible reasoning becomes load-bearing changes across model releases.
Limitations
Ultimately, we want to understand whether models can carry out seriously harmful actions without being detected by monitors. Planning without CoT is one capability that could contribute to this: a model could compare strategies and anticipate consequences without exposing that reasoning to a monitor. We track this capability as a potential early warning signal.
But models may improve at different kinds of latent reasoning at different rates, and exploiting an immediate opportunity to cause harm may require different capabilities from solving our planning tasks. In a no-CoT evaluation on a subset of BashArena, models completed 61.6% of harmful side tasks while completing the legitimate cover task in only 0.29% of attack trajectories. These results show that some harmful tasks are already achievable without CoT, but do not establish how much internal planning they required. Models were explicitly prompted to attack, which also limits what we can infer about their behavior under ordinary instructions.
We therefore should not treat weak performance on this benchmark as evidence that a model cannot cause serious harm. We nevertheless expect planning to matter for sustained attempts to circumvent safeguards, where a model may need to anticipate later checks, coordinate several actions, and account for how those actions affect its chances of detection. That makes no-CoT planning worth tracking alongside direct evaluations of harmful behavior and monitor evasion, while recognizing that the relationship between these capabilities remains an open question.
Discussion
No-CoT planning has improved substantially, particularly with Astra, but performance still deteriorates as planning horizon increases. Prompt repetition and filler tokens improve Astra’s results even at deeper horizons, showing that baseline scores understate what the model can achieve without writing out intermediate reasoning. These findings leave an important distinction between demonstrating some capacity for latent planning and performing it reliably. The RL result suggests that it is at least partly trainable. Direct optimization improved performance, so future gains may depend less on broad capability scaling than on training methods that encourage computation to remain latent.
One possible explanation for this divergence is that latent planning requires models to maintain and update intermediate state internally rather than in a visible trace. Errors may therefore compound across steps. A model may represent the early stages of a plan correctly but lose track of constraints, costs, or state information as the horizon grows. This remains a hypothesis, however; the benchmark does not distinguish failures of state tracking from failures of search or representation.
No-CoT planning should therefore be treated as a moving capability boundary rather than a fixed limitation of AI systems. The present benchmark offers one way to track that boundary, but its longer-term value will depend on whether it can distinguish generalizable improvements in latent planning from environment-specific adaptation. Identifying which training and architectural changes move the boundary, whether those gains transfer to unfamiliar tasks and longer horizons, and whether they reduce the practical coverage of chain-of-thought-based oversight are promising directions for future work.
The author would like to thank Olivia Watkins, Miles Wang, Yong Zheng-Xin, Jason Wolfe, Micah Carroll, Bowen Baker, Zora Che and Leon Maksin for valuable feedback on drafts of this post.