tl;dr I have tested Astra's ability to complete various long multi-step tasks without using its chain-of-thought, and found that the number of sequential steps is a poor predictor of task success. Instead, Astra's ability to solve a task seems to correlate more strongly with what I will call the speculative depth of the task. Based on experimental results, it seems less likely that Astra solves sequential no-CoT tasks purely by reasoning step-by-step in latent space. Instead, Astra appears to do some form of speculative reasoning, where intermediate results are guessed based on heuristics, and then iterated upon in parallel until they become self-consistent. This allows many multi-step tasks to be solved with far fewer serial steps than naively seems possible, especially if the initial guesses are good. Other LLMs also appear to behave like this, but to a much smaller degree.

In my previous post, I discussed how certain KV-cache sharing schemes may lead to long opaque serial paths, and introduced the LatentMathBench microbenchmark, which measures an LLM's ability to solve tasks that require many consecutive steps without using their chain-of-thought. Several other users have done more extensive no-CoT reasoning benchmarks based on more varied (and often more realistic) tasks, e.g. RohanS, Neel Nanda, and Dylan Xu, SebastianP and Alek Westover. While these benchmarks all see significant improvements in Astra's no-CoT abilities compared to previous models, LatentMathBench seems to be an outlier in terms of how large the gap is between Astra and any other LLM (roughly 3-4x). I wanted to better understand what causes this huge gap, so I experimented with various different types of tasks in an attempt to understand what is driving this capabilities gap.

Shortcuts

I repeatedly stumbled upon tasks where Astra would seemingly perform absurd numbers of sequential steps without CoT, such as this 500-step math task with one variable. However, upon closer inspection all of these tasks turned out to be flawed in some way that made it possible to predict the right answer without doing all the steps. For example:

• In the 500-step math task linked above, there are many integer division operators that map multiple inputs to the same output. This is less of an issue in the default math-4 task (which has 4 variables), but with only one variable in play, this reduces the number of possible states at various points and hides errors made in earlier parts of the calculation. For the linked example, you can pick just about any number, then do just the last 20 steps, and get the correct result.
• In tasks with long chains of addition and subtraction, but no multiplication or division, it is possible to calculate the result in parallel using tree reduction strategies. It is plausible that LLMs would have learned such strategies, since sums of many numbers will occur naturally in pretraining data (e.g. itemized bills followed by a total amount).

This led me to add two new task types to LatentMathBench, designed to be hard to shortcut: swap-N, which replaces the calculations with simple swaps^[1], and bitwise-N, which mixes addition, subtraction and bitwise XOR to prevent tree reduction strategies. Both variations produced lower scores for all tested models, but Astra was still the clear outlier, outperforming all the other models by roughly a factor of 3.

Boolean circuits

At some point I thought that Astra's internal calculations might work somewhat like an analog computer, which accumulates error with each operation, such that long chains of calculation become unreliable due to accumulating error. In an attempt to avoid that issue, I created a benchmark task based on boolean circuits, which should be less sensitive to accumulated error. The bool-N tasks look like this:

All steps are of the form a = a xor (b and/or c). The generated tasks all have the following properties:

• Each step is reversible, so no information is lost. This means an error at any point in the calculation will propagate to the final result (unless it is canceled by another error).
• Each step depends directly on the result of the previous step (the destination of the previous step is one of the inputs of the and/or operator of the next step).
• Each step contains two sequential boolean operations, so an N-step task requires 2N sequential boolean operations.
• The LLM is required to output the final value of all variables, in order of last assignment. This prevents step-by-step reasoning in the output and makes guessing less effective.

The results were fascinating, but for a completely different reason than I expected:

(I define "50% success horizon" as the maximum number of steps for which the rate of success is at least 50%.)

Two strange things are happening here. One, Astra is reaching absurd numbers of steps: 54 steps corresponds to 108 sequential boolean operations! Two, Astra's performance becomes significantly better when there are more variables, while the other LLMs are relatively insensitive to this. Why is that?

A possible explanation is that and/or operations can be short-circuited. For example, if you want to evaluate a and b, and you already know a is false, then you don't need to calculate b: the result is always false. Many programming languages exploit this to skip unnecessary calculations. When there are more variables, the time between two assignments to the same variable increases, which means there are more opportunities to short-circuit operations several steps ahead. Also, with more variables, many of the early steps in the chain will be obviously short-circuited by the initial value of other variables that have not yet been overwritten. Note that the reversibility of the entire a = a xor (b and/or c) step does not preclude short-circuiting the and/or of part of that step - these are independent properties.

Another possible explanation is that the outcome of and/or operations is biased: given random inputs, a or b has a 75% chance of being true, while a and b has only a 25% chance of being true. This enables some statistical tricks where the result can be predicted to some degree without actually doing all the steps. I expect that this effect would be strong for randomly generated boolean circuits, but this specific test uses steps of the form a = a xor (b and/or c), which keeps a unbiased (due to reversibility), so this explanation seems a bit less likely here.

We can test whether some combination of short-circuiting and statistical bias is responsible by replacing the and/or operators with xor. This makes the whole task 'linear' (in the mathematical sense), which means it can theoretically be tree-reduced (via binary matrix multiplication), but it seems unlikely that an LLM would use such a specialized technique for such an obscure artificial test. Indeed, all LLMs seem to find the xor-only version of this test significantly harder:

Astra's 50% success horizon for bool-26 drops from 54 to 11. The improvement when more variables are added is now also much smaller, which is a strong indication that a combination of short-circuiting and statistical bias was responsible for both effects.

Notably, the success horizon of the other LLMs also decreased significantly, even for older models like GPT-4. That suggests all LLMs were using some form of this trick.

But how?

Epistemic status: Very speculative!

How did Astra learn these task-specific tricks, which only work on these highly artificial tests that are unlikely to appear much in pretraining? For that matter, how did the much older GPT-4 learn it? It seems implausible that LLMs would learn so many complicated tricks that work in so few cases.

I think something much more general is happening here. For boolean circuits specifically, there exists an algorithm called speculative evaluation (or optimistic evaluation), which can evaluate boolean circuits in fewer sequential steps than the logical depth would predict, at the expense of doing far more parallel steps. It works like this:

• Convert the circuit into a graph representation. In a programming context, this is equivalent to converting the steps to static single-assignment form.
• Guess the value of all intermediate results using some heuristic.
• Evaluate all boolean operations in parallel based on their assumed input, and update the resulting outputs.
• Iterate until this converges to a consistent result.

For a circuit with logical depth D, this requires at most D iterations, though it may also require fewer iterations, especially when the initial guesses are good. This algorithm is highly inefficient in terms of the total number of operations it must evaluate, but very efficient in terms of the number of serial operations it requires. And this algorithm works on a very wide range of situations: it can take advantage of short-circuiting, but also statistical bias, convergent paths (such as in the 500-step math example), and many other shortcuts, all with a single algorithm, without needing special understanding of the specific task. Is it possible that LLMs are doing something like this?

Speculative reasoning

The structure of the boolean circuit speculative evaluation algorithm fits surprisingly well into the structure of a transformer.

The first step, converting the circuit into a graph, can be performed in parallel by the attention mechanism: it amounts to "find the last occurrence of a = before this token", which is the kind of thing LLMs do all the time. But note that this only works when the operations are presented in the correct order, as in this benchmark.

The second step, guessing the intermediate results using heuristics, is again something transformers (and neural networks in general) do extremely well. It's not particularly difficult for an LLM to learn from pretraining that the result of a and b is more likely to be false than true.

The tricky part is the iterative step. In a standard transformer architecture, this iteration would have to happen in consecutive layers, which means multiple of those layers need to have circuits built into them for the exact same operations (which is feasible for simple boolean operations, but becomes more difficult for arithmetic operations). Given a finite number of layers, each of which probably only implements a subset of the required operations, the LLM's ability to execute this iterative part of the algorithm is limited.

This is where Astra's supposed use of recurrent depth might change everything. Suddenly, the iterative structure is built directly into the network! The duplicate circuits required to converge to a solution are now present automatically! In fact, if the recurrence depth is adaptive, either via some trained mechanism or by iterating until the activations converge to a stable value, the "iterate until convergence" aspect of the algorithm can be baked into the network itself!

Is this why Astra overperforms on these types of tasks? Given how little we know about the architecture of Astra, we can only speculate, but it does seem like a plausible explanation for the experimental results I'm seeing. It seems a lot more plausible than my earlier theory that Astra is doing some form of per-token recurrence via KV-cache sharing. That theory might have made sense to explain latent reasoning during output generation, but this benchmark measures ability to reason during input processing (prefill), which is generally done in parallel. Introducing per-token recurrence during prefill, even with very limited depth, would lead to significant implementation challenges^[2]. Note that this does not exclude KV-cache sharing schemes during output generation, which is inherently sequential, and where KV-cache sharing is appealing since it reduces KV-cache size, even if there is no performance benefit.

Testing task success rate vs speculative depth

A natural way to test this theory is to implement the speculative evaluation algorithm for boolean circuits, measure the speculative depth of the circuit based on how many iterations are required to obtain the correct result, and then test whether task success correlates with speculative depth.

Note that this definition of speculative depth is highly dependent on the chosen algorithm. My implementation of speculative evaluation is rather simple: I just use the initial values of the variables as the initial guess for all intermediate results of that variable, and then iterate until convergence. Note that I treat a xor (b and/or c) as one operation, not two. On average, a bool-N task (which has N steps) seems to have a speculative depth around N/2.

I generated test cases with a specific number of steps and a specific speculative depth, and measured task success rate for each combination, which produces the following result^[3]:

The task success rate is clearly strongly correlated with speculative depth, in fact speculative depth alone seems to be a better predictor of task success than the number of steps alone. Based on these results, I now think the KV-cache sharing hypothesis I presented in my previous post is less likely to be true for Astra, especially during input processing (prefill), though of course these results do not disprove it either.

Open questions

Exact form of the speculative algorithm: I think the benchmark results support that some form of speculative processing is happening in Astra, but it doesn't necessarily have to be the exact speculative evaluation algorithm described here. There are many different ways an LLM could converge to a coherent solution with fewer steps than the logical depth predicts, using various combinations of heuristics, forward propagation of inconsistencies, parallel testing of multiple candidates (boolean variables have only two possible values, so many calculations could be precalculated for multiple possible inputs, saving time once the correct input becomes known), ...

Order dependence: The speculative evaluation mechanism presented here would depend on operations being in the correct order. Explicit conversion to static single-assignment, followed by random shuffling of the operations, could reveal how order-dependent this mechanism is.

Error-structure analysis: When Astra produces incorrect results, in what way are they incorrect? Studying these errors might produce more insight into the underlying algorithm. I intend to focus more on this in the future.

Impact of filler tokens: Several no-CoT Astra benchmarks, most notably this one, find that adding filler tokens increases performance on some no-CoT tasks. I have not yet explored this in LatentMathBench.

Latent thinking during output generation: The current benchmark tests almost exclusively how well LLMs can reason during input processing (prefill). If a recurrent path exists via KV-cache sharing, it is much more likely to show up during output generation instead. This could be explored by instructing the LLM to generate some predefined number of filler output tokens (e.g. dots) before answering.

Sensitivity to architecture: To what extent are these results specific to the Astra architecture? Is the apparent increase in speculative reasoning capabilities a consequence of recurrent depth, or is it merely a consequence of increased effective depth itself, such that it would appear in any LLM architecture with sufficient layers? Testing with open-weights recurrent depth/looped transformer models such as Huginn and Ouro may produce new insights.

Relation to adaptive recurrence depth: Is this result a consequence of an adaptive recurrence depth scheme, either as a trained mechanism, or through some kind of 'iterate until activations converge' strategy?

Relation to hallucination rate: An obvious problem with speculative reasoning strategies implemented with a finite number of iterations is that if the result does not converge within the allocated number of iterations, the LLM may output an incoherent result, resulting in hallucinations. To avoid that, the LLM would have to track its own uncertainty, or measure the convergence status of the final result, to decide whether it should produce output or perform further reasoning via chain-of-thought. Anecdotally, Claude models appear to behave like this in my benchmark: when the number of steps is limited, they adhere to the system prompt which tells them not to use chain-of-thought, but when the number of steps becomes large and their error rate increases, they start violating this rule and reason anyway to get the correct result (the benchmark counts such results as incorrect). Astra does not do this: it adheres strictly to the system prompt and never uses chain-of-thought, even if that results in completely incorrect results.

Implications for chain-of-thought monitorability: If these results generalize beyond artificial test cases like boolean circuits, our intuitive notion of how many steps are required to covertly plan misaligned actions may be inaccurate. What we view as a 20-step plan might require fewer speculative iterations, particularly when good heuristics are available to predict the intermediate results, and multiple candidate strategies can be evaluated in parallel. It is not clear how one would measure the irreducible depth of realistic real-world tasks.

1. ^{^}

   In theory, swap-N is tree-reducible, but this requires an extra level of indirection which may be unnatural to LLMs since it doesn't fit the KV-cache lookup mechanism. In practice, LLMs seem to find this task more challenging than math-N, despite not requiring any calculations.

2. ^{^}

   This is mostly a limitation of today's GPU architecture, not a fundamental computational limit. With custom ASICs, very limited depth per-token recurrence becomes a lot more feasible. OpenAI could hypothetically have added such functionality to their Jalapeño ASIC, but given the timeline it seems unlikely to me that this would have been used in Astra. (I am an ASIC designer by trade.)

3. ^{^}

   Limited to 10 trials per test case for cost reasons, so results are somewhat noisy.