The poetry case really stuck with me. Claude’s clearly planning rhymes ahead, which already cracks the “just next-token” intuition about autoregressive models.
But maybe it’s more than a neat trick. What if this spatial planning is a core capability—like the model’s not just unrolling a string, but navigating a conceptual space toward a target?
One could test this by checking how often similar planning circuits pop up in multi-step reasoning tasks. If it’s building a rough "mental map" of where it wants to land, that might explain why bigger context windows boost reasoning so much. Not just more data—more room to plan.
Has anyone tried prompting or tracing for this directly?
The poetry case really stuck with me. Claude’s clearly planning rhymes ahead, which already cracks the “just next-token” intuition about autoregressive models. But maybe it’s more than a neat trick. What if this spatial planning is a core capability—like the model’s not just unrolling a string, but navigating a conceptual space toward a target? One could test this by checking how often similar planning circuits pop up in multi-step reasoning tasks. If it’s building a rough "mental map" of where it wants to land, that might explain why bigger context windows boost reasoning so much. Not just more data—more room to plan. Has anyone tried prompting or tracing for this directly?