Epistemic status: We found useful uncertainty signal in three reasoning models. All experiments were run on three seeds and are fully reproducible. Whether the same approach can become a broader tool for monitoring models is still an open question. Lingo might include “intend” or “conscious” and has to be taken with a grain of salt.
TL;DR:
We provide an uncertainty estimation method for reasoning models. Building on the J-lens, we construct four orthogonal directions in a model’s hidden-state space: ambiguity, incompleteness, conflicting evidence, and general uncertainty. Together, these directions span the U-space. As the model reasons, we measure how closely its hidden states align with each direction. Paired with the average entropy of the model’s next-token probability distributions across reasoning tokens, this yields the U-lens. Its readout helps us spot answers that are more likely to be wrong. Surprisingly, just counting the tokens in a model’s reasoning gives us a good indicator of whether the final answer is right or wrong. Wrong answers tend to come with longer reasoning traces. To mitigate this potential shortcut, we also compared traces of similar length, and the U-lens holds its ground across three models and four benchmarks. Notably, our approach is label-free and requires just a single forward pass, while many other methods require training or multiple passes. The U-lens also held up in our out-of-domain analysis across benchmarks better than many trained probes. Finally, early tests on emotions and reward-hacking responses hint that similar readouts could help model harnesses flag suspicious behavior. We’ve open-sourced the code.
Introduction
Birth of the idea
Earlier this year, Anthropic connected the global workspace theory to large language models (LLMs). They showed that a model’s hidden state includes concepts that are not necessarily verbalized in the output but nonetheless used during intermediate reasoning. Intrigued, we wanted to put this new form of semantic readout to work on something with direct real-world applications. Coincidentally, one of us had just stumbled upon a paper by Kumaran et al. that discusses a language model’s metacognitive awareness of its own uncertainty. The authors found that the output probabilities a model emits during the autoregressive generation of its answer and the self-reported uncertainty that it expresses when explicitly asked for it (e.g., “How certain are you about this answer?”) contain complementary but not fully overlapping signals of its confidence. So we wondered (a) can we combine these two types of uncertainty signals in one estimator, and (b) can we improve the capture of the self-reported uncertainty by leveraging the Jacobian readout introduced in the J-lens?
Contributions
We built the U-space from four kinds of uncertainty: ambiguity, incompleteness, conflicting evidence, and general uncertainty. The alignment of a model’s hidden state with these basis directions allows us to watch different forms of uncertainty come and go as the model reasons.
We paired the interpretable U-space readout with the average entropy of the model’s token probabilities to get a scalar uncertainty score, the U-lens. It combines the two uncertainty signals discussed in Kumaran et al.
Across three models and four benchmarks, the U-lens was better at separating right from wrong answers than 12 other methods we tested. It held up even when we compared reasoning traces of similar length to remove a potential length confound.
We pushed one model along the uncertainty directions and made it hesitate over simple questions in different ways. The directions appear to affect the model's behavior rather than just describe it.
We tried the same construction beyond uncertainty. In early tests, it read emotions in comments and distinguished reward-hacking responses from genuine ones.
Background
When an LLM is unsure, it rarely says so. It usually produces a plausible but hallucinated answer. This uncertainty is intrinsic to the model. It is the dispersion of the model’s predictive state for a given query, and uncertainty quantification (UQ) estimates it. The uncertainty signal can be read from the model’s output or from its internal computation. For example, we can look at the probability the model assigns to its answer tokens or at how spread out its next-token probabilities are. We can also ask the model multiple times and compare whether the sampled answers mean the same thing, or ask it how likely its own answer is to be true. That last option is tempting, because we can apply it to open- and closed-source models without any interventions in the computation, but self-reported confidence is often overconfident. It also asks the model to turn its internal uncertainty into a string of tokens, which could be a noisy transformation.
Another line of work looks at the internal states of a language model and tries to put what it finds into a form we humans can understand. In a transformer, layers pass information through a shared residual stream. The vector in that stream at a particular layer and token position is what we call a hidden state. Each block reads from the stream and adds its contribution back, until the final state is used to predict the next token. The linear representation hypothesis suggests that at least some concepts show up as directions in these hidden states. U-space makes this bet for uncertainty. If different kinds of doubt have distinct directions in the residual stream, we can look for them without asking the model to put them into words. Several approaches try to read these intermediate states before the model produces its final output distribution. The logit lens reads a state as if it were the final one, skipping the remaining layers and applying the unembedding matrix directly. The tuned lens learns an auxiliary linear map before applying that matrix. The J-lens gets its map from the model itself. Its Jacobian asks how a small change in an intermediate hidden state would affect the final state of the current and later tokens, after passing through the remaining layers. Averaged across many contexts, this lets us read out concepts the model could put into words even when it does not.
where is the hidden state at layer and token position is the final layer, and the expectation runs over prompts and over the current and all later positions . is the unembedding matrix and the gain of the final RMSNorm. The R-lens improves early-layer readouts by adjusting the backward pass to reduce accumulated errors.
These lenses show us what the model represents during its computation, but they do not provide a measure of its uncertainty. The J-lens does give each vocabulary token a direction in the residual stream, though, and this gives us a place to start. First, find words that express different forms of uncertainty, then derive corresponding directions in the residual stream, and finally check how much the hidden states align with these directions.
Constructing the U-space
Finding the right basis
Before building the U-space, we need to decide what kinds of uncertainty should build its basis directions. For the U-space, we want categories that make intuitive sense, have roots in the literature, and are few enough that we can still tell them apart cleanly. We compare several candidates. The aleatoric and epistemic split separates model-independent randomness from a lack of knowledge of the underlying input-output relationship. Klir and Wierman distinguish fuzziness (where does “tall” begin?), nonspecificity (which of several possibilities is right?), and discord (what if the evidence disagrees?). Prince et al. distinguish four simple ways of softening a claim: making it less exact (“sort of”), making a number approximate (“about twenty”), signaling personal uncertainty (“I think”), or shifting responsibility to another source (“according to the report”). AmbigQA looks at ambiguity in questions: which event or entity is meant, what property or type of answer is sought, whether the answer depends on time, or whether several questions have been rolled into one. Finally, we test a taxonomy inspired by Parsons. It groups uncertainty into four forms: ambiguity, incompleteness, conflicting evidence, and general uncertainty. We compare it with the other candidates on how intuitive their categories are and whether they strike the right balance, namely, enough to distinguish useful kinds of uncertainty, but few enough that we can still separate their signals and visualize them in 3D. We also test quantitatively how well each taxonomy helps us spot wrong answers.
Considering all criteria, we settle on the four categories inspired by Parsons: ambiguity, incompleteness, conflicting evidence, and general uncertainty. The last category leaves room for uncertainty that was not captured by the others, such as an unreliable source.
Author note: We could also get anchor words from the model itself. With contrastive prompts, the J-lens can read out words associated with each category. We stuck with published word lists to avoid depending on carefully crafted prompts, although words extracted from the model improved uncertainty prediction in preliminary experiments. See Appendix B for examples of such contrastive prompts.
From words to residual-stream directions
Having four categories is one thing. Finding them in a model’s hidden states is another. We start with published lists of uncertainty and certainty words. A lightweight sentence encoder sorts all uncertainty words into our four categories using short definitions of each category. For instance, the word “contradictory” is assigned to the conflicting evidence category, while “dubious” is assigned to general uncertainty (see Appendix C). Next, we pull each word back through the J-lens.
where
: a one-hot vector that picks out word in the vocabulary,
: the unembedding matrix as above,
: the gain of the final normalization layer (RMSNorm) as above,
: the J-lens Jacobian, which describes how the final hidden state responds to changes in the hidden state at layer averaged over generic prompts,
: the resulting direction for word in the hidden states of layer
This tells us which direction in a chosen layer’s residual stream would raise that word’s output score. The word is now a direction we can look for inside the model, rather than something we need to find in its written answer. For each category, we average all corresponding uncertainty-word directions and subtract the average of all certainty-word directions. This helps cancel out what the two groups have in common and gives us a direction pointing toward that specific kind of uncertainty.
where and are the uncertainty and certainty words of category and scales a vector to unit length. We then make the four directions perpendicular and equally long. Together, they span the U-space.
This symmetric (Löwdin) orthonormalization keeps each column of as close as possible to its category direction . This takes no examples of correct or incorrect answers, requires no training.
Reading uncertainty from the U-space
With the four basis directions in place, we can follow the model as it reasons. At each step, we project its hidden state onto the U-space. This gives us four numbers, one for each kind of uncertainty, which we can plot across the reasoning trace.
where is the hidden state of reasoning token at the readout layer, the columns are the four U-space directions, and is the signed alignment with category We keep signals from different categories from canceling each other out. If the state aligns with conflicting evidence but points away from ambiguity, the conflicting-evidence signal should still count. We achieve this using the elementwise softplus function:
where , is applied on each coordinate, and the gain is fixed by the number of categories. A negative alignment with any of the axes shrinks toward zero instead of subtracting from a positive one. is a single score that rises when the hidden state points toward one or more uncertainty directions within the U-space (positive combinations of those directions form a cone within the U-space) and represents the verbalizable uncertainty referred to at the beginning. To get a single readout for the whole reasoning trace, we look at the hidden state at the end-of-thinking marker. By then, the model has processed all its reasoning, and we have one consistent point to inspect, no matter how long it took to get there (this will become important in the “Evaluating the U-lens” section, where we show that many other methods get strongly influenced by the length of the reasoning trace). This concludes the extraction of the verbalizable uncertainty signal that we discussed at the beginning. While the model does not explicitly report its uncertainty through its generated tokens, the signal is extracted from its residual stream. As also discussed there, we pair this verbalizable readout with a distributional signal from the model’s next-token probabilities. At each reasoning step, we measure how spread out those probabilities are, then average over the trace, so longer reasoning does not automatically mean a higher score.
where is the probability the model assigns to token of its vocabulary at the reasoning step and is the length of the reasoning trace Multiplying that average by the U-space readout gives us the U-lens uncertainty score.
where is the position of the end-of-thinking marker. A higher signals an answer that is more likely to be wrong.
Author note: We multiply the two signals so the U-space readout can adjust predictive entropy without a fitted scale. A more carefully tuned combination might spot more wrong answers, but would usually need calibration examples, which we wanted to avoid.
Evaluating the U-lens
The U-lens gives us two things to examine: four uncertainty signals we can follow token by token, and an overall score we can later compare with whether an answer is right. We test it on three open-weight reasoning models: Gemma-4-31B-it, Qwen3.5-27B, and Magistral-Small-2507. We fix the readout layer in advance at two-thirds of each model’s depth: layer 40 of 60, 42 of 64, and 27 of 40, respectively. This is a simple way to land in the range where the J-lens reads intermediate representations well, after the earliest layers but before the final layers become dominated by the output token. It also keeps us from picking a favorable layer separately for each model. We use released J-lenses for Gemma and Qwen. For Magistral, we estimate one from 100 unlabeled WikiText-103 prompts using the reference implementation. Before checking whether the score predicts wrong answers, let’s see how it reads the different uncertainties from a single response to the question: “Was Hamlet left-handed, and did he love Ophelia?”
At first, the verbalized answer and the U-lens agree. Shakespeare never specifies Hamlet’s handedness, and incompleteness lights up. Then the answer becomes much firmer: “Yes, Hamlet did genuinely love Ophelia.” The readout tells a less settled story. It briefly shifts toward ambiguity at the word “play,” then conflicting evidence takes a larger share and remains prominent as the answer mentions Hamlet’s cruelty. General uncertainty also lights up at the apostrophe in “father’s.” (The lens might be picking up a more local kind of uncertainty, perhaps about grammar, but we did not test this further.) Another example can be found in Appendix D.
The Hamlet example shows what the readout picks up in one answer. Now, we move to the quantitative evaluation and test whether the U-lens also helps to flag potentially erroneous answers across an entire set of questions. We test this on 1,000 questions each from MMLU-Pro, Omni-MATH, SuperGPQA, and TriviaQA. They cover broad reasoning, Olympiad mathematics, graduate-level knowledge, and factual recall. We sample three responses per question from each model. AUROC and AUPRC tell us how well high uncertainty scores pick out wrong answers. AURC tells us how the error rate changes as we set aside answers flagged as uncertain. One important phenomenon when evaluating uncertainty estimators is that incorrect reasoning traces tend to be longer than correct ones. Indeed, across our four benchmark comparisons, the median length of incorrect traces is between 1.33 and 4.79 times that of correct traces, although their length distributions overlap considerably. This creates a potential confound. A method might appear effective at predicting errors simply because it assigns higher uncertainty scores to longer responses, rather than identifying uncertainty in the model’s reasoning. We therefore score each method twice. First, we evaluate all responses together, allowing reasoning length to contribute whatever signal it carries. Then we compare responses of similar length: we split each evaluation set into ten groups by token count, calculate the same metrics within each group, and take an average weighted by group size. Comparing the two scores shows how much of a method’s performance is tied to response length.
We compare the U-lens against a range of methods that use different strategies to extract uncertainty signal. Generation length simply counts the number of reasoning tokens and uses the total length to predict whether an answer is correct. MSP, Predictive Entropy, Maximum Entropy, Mean NLL, Self-Certainty, and DeepConf draw on the token probabilities during autoregressive generation. P(True) asks the model plainly to judge the certainty of its answer in a second pass, TokUR perturbs its weights across several passes, and Semantic Entropy and SentenceSAR compare multiple sampled answers. We also include Feature-Gaps, which learns a hidden-state direction from labeled examples.
Length alone is a remarkably strong baseline. At 68.6% AUROC, it nearly matches the supervised Feature-Gaps method at 68.8%. But when we compare answers of similar length, its score falls to 52.4% (close to random guessing). TokUR, the strongest baseline on unadjusted AUROC, loses 9.4 points. The U-lens moves from 71.1% to 69.2%, but still claims the highest length-controlled AUROC. This suggests it picks up more than just how many tokens the model emitted to answer the question.
The workspace signal and entropy help each other
The U-lens combines two signals. The first is the verbalizable workspace signal, which we called in the construction of the U-space, and which is read from the hidden state at the end of reasoning. The second is the distributional signal, which is measured by the average predictive entropy of the next-token distributions during reasoning. We now want to test whether they capture the same uncertainty signal, or whether each adds something the other misses. We compare each signal on its own and with their combination in the U-lens score.
The two signals help in different ways depending on the model. For Gemma, the workspace signal and predictive entropy work about equally well on their own. Qwen gets more from entropy, while Magistral gets considerably more from the workspace signal. Combining them works better than either one alone for all three models.
Do the directions affect generation?
Our approach thus far shows that U-space directions can be used to identify uncertainty-related information in hidden states and thereby predict incorrect reasoning. However, this does not tell us whether these representations actually influence the model’s behavior. To determine this, we perform activation steering experiments by modifying the model’s hidden states along individual uncertainty directions and observing how its generation changes. For a hidden state we apply . Here, is the direction corresponding to one uncertainty category, while controls the strength of the intervention relative to the hidden state’s norm. This makes intervention strengths more comparable across layers and positions. We conduct the experiments on Qwen3.5-27B, steering blocks 32–54 in two settings:
Prompt-level steering: We modify activations during the prefill phase with thinking disabled. Asking the question “What is 2 + 2?” allows us to observe how each uncertainty direction affects a response that should ideally require little interpretation or uncertainty.
Reasoning-level steering: We intervene during a 50-token window inside the thinking process, beginning after a fixed prefix that the model has already generated.
For the exemplary question “What is 2 + 2?” the model still generates “4” at moderate intervention strengths, yet it introduces peculiar reasoning detours. Ambiguity steering leads the model to question whether the prompt itself is valid, and steering conflicting evidence causes the model to recall literature references where “2 + 2 = 5.” When steering within the reasoning process, amplifying the ambiguity direction causes the model to introduce an additional step, namely “Review for Ambiguity: Is there any trick?” before concluding with the arithmetic problem.
These experiments show us that the U-space directions are not merely useful for reading hidden states and that intervening along them can also change the model’s subsequent behavior. The evidence at this stage is preliminary. For now, the results provide an initial check that complements the general U-space methodology. They do not establish that these four axes constitute distinct causal mechanisms of uncertainty.
Does this approach generalize to other domains?
The recipe behind U-space starts with contrasting words, so we tried it beyond uncertainty. On GoEmotions, we built directions for anger, sadness, fear, and general negative feeling using words such as “angry” and “calm.” All four picked up their target in human-labeled Reddit comments, even when the comments contained none of our chosen words.
The sadness direction, for example, could distinguish sadness from other negative emotions, suggesting it captures a more specific semantic property beyond mere negative sentiment. We also tried School of Reward Hacks, where each task comes with one response that does what was asked and another that exploits the stated scoring rule, for example, by stuffing in keywords. This time, we let the model suggest the anchor words from examples, then used those words to construct a semantic direction.
The resulting direction separates unseen responses and cheat methods across all three models. The hacked responses are primarily decoded as bizarre or fake rather than an intent to cheat, but this is a clear signal nevertheless. The way a model expresses a certain semantic property internally might not one-for-one align with how humans would.
Does U-lens hold up when the domain changes?
A trained uncertainty estimator might learn something specific to the questions it was trained on. We tested this by fitting five hidden-state uncertainty probes on labeled answers from one benchmark, then applying them unchanged to the other three. We repeated this across MMLU-Pro, Omni-MATH, SuperGPQA, and TriviaQA. The U-lens uses no correctness labels, so there was nothing to refit.
The scores are length-controlled AUROC × 100. For each model and trained method, we fit a probe on one benchmark and test it on the other three. We repeat this with each of the four benchmarks as the training set, giving us twelve transfers to average. The U-lens needs no benchmark-specific fitting, so we simply average its results across the four benchmarks. The trained probes often did well on their own benchmark and lost ground when the questions changed. Mass-mean was the most stable of them, but the U-lens still came out ahead of the strongest transferred probe for each model. The margin on Gemma is smaller than on the other models. Across these four benchmarks, the result suggests that our uncertainty directions are less tied to a particular set of questions.
Limitations
Our results suggest that U-space provides a useful and novel way to detect and interpret internal uncertainty, yet several limitations remain. Firstly, the uncertainty categories are not exhaustive. Our four categories were chosen for their interpretability, but they do not necessarily reflect how models organize uncertainty internally. Furthermore, the method depends on the quality of the underlying lens. U-space relies on the J-lens to map semantic concepts back into the residual stream, meaning its effectiveness is contingent on how accurately the lens represents them. This could limit applicability to models where the J-lens is less reliable, such as smaller models. Although preparing a lens can require estimating an average Jacobian, it is relatively inexpensive and a one-time procedure involving backward passes through a frozen model, so no parameter optimization or task-specific labels are needed. Related to this is the limitation that applying the U-lens requires access to internal model states. The method requires hooks during generation to extract the hidden states and token-level predictive distributions to calculate the workspace signal and predictive entropy. To extend this to closed-source models would depend on providers exposing the required internal information or implementing the method themselves. Finally, we have not evaluated the method adversarially, yet. Our experiments focus more closely on standard reasoning benchmarks rather than examples deliberately constructed to expose weaknesses. Consequently, we do not yet know how robust the method is to adversarial prompts or attempts to produce misleading uncertainty signals. We leave this as an interesting continuation of our work.
Where this leaves us
We began by asking whether a model’s next-token probabilities and its verbalizable uncertainty could tell us different things, and whether we could read the latter without asking for a self-report. Our results suggest yes on both counts. The U-lens combines a workspace readout with average predictive entropy of the reasoning trace, and each signal adds something that the other misses. Together, they beat every single one of the twelve baselines we tested. We kept the method deliberately simple, without fitting a scale on labeled examples or using handcrafted contrastive prompts for the anchor words, so there is room to tune it for particular uses. The same way of building directions also showed promise beyond uncertainty. Our emotion and reward-hacking results were encouraging.
Before we close, let us move from uncertainty quantification to a broader question about oversight, or in other words, the question of whether we could see signs that a model is heading toward a harmful action. We used the J-lens to find uncertainty in a model’s computation without asking it to say what it was unsure about. Could we look for signs of a harmful plan in the same way? If we abuse the global workspace analogy, we could venture to consider the verbalizable representations in a model’s residual stream as the model’s “thoughts” it considers before answering. This leads us to a slightly brazen question about our own thoughts. Can a human craft a sophisticated, harmful plan without ever thinking about its harmfulness? Probably. But you would imagine that it becomes harder to carry out. We wonder if something similar could hold for models. A model can learn to hide misbehavior in its written reasoning, but we suspect a misaligned plan may still leave traces in its verbalizable workspace. Checking that space across tokens and layers might make those traces harder to hide. Our U-space directions are linear, though, and recent work on concept manifolds shows that some concepts form curved structures that a single direction captures only in part. We think combining checks throughout the reasoning with ways to read such structures is a promising question for oversight. Consider this pothead hypothesis as food for thought, nothing we tested. We do not know whether a misaligned plan must pass through the verbalizable workspace, or whether a model could easily learn to route around these checks. But even a monitor that makes evasion just a little bit harder might buy us time to develop better oversight.
Project page:s2lab.org/U-Space, with interactive token-level examples, and a video series that walks through the method.
Code:github.com/s2labres/U-Space (BSD-3-Clause). A single config (configs/reproduce.yaml, run with ./run.sh) reproduces the U-lens and single-pass baseline results for all three models, four benchmarks, and three generation seeds.
We are currently packaging the U-lens to make it as easy to use as possible, because we hope it can be incorporated into all kinds of settings and systems.
Appendix A: Taxonomy ablation
Length-matched U-lens AUROC (%) when the four U-space categories are replaced by other taxonomies. The pool of potential anchor words, the lenses and the readout layer stay fixed. The cells are the means over three generation seeds. The Macro is the mean over the four benchmarks ± the sample standard deviation over seeds. The best value per model and column is marked in bold. Source: paper, Appendix I.
Appendix B: Contrastive prompts
One content-matched prompt pair per category (12 pairs per category in total) and the 15 words that the J-lens decodes from the average difference between the uncertain prompt and its resolved counterpart, on Qwen3.5-27B. The highlighted spans mark where the two prompts differ, and the numbers shaded in gray are cosine similarities. The bold words were matched to the category by hand for visualization purposes. The non-English tokens are translated, with their original language in superscript. Source: paper, Appendix C.
Appendix C: Anchor words
Uncertainty words: all 61 cue words of Chen et al. (2018), as assigned to the four categories by the lightweight sentence encoder. Certainty words: of the 348 single-word entries of the Rocklage et al. (2023) certainty norm, the ten per category whose pulled-back directions align best with their category average, averaged over the three models. The counts in parentheses are before dropping words that a model splits into several tokens.
Appendix D: A second token-level example
The same readout as for the Hamlet question, for a riddle-like prompt about Michael Jackson’s moonwalk (Qwen3.5-27B, 10-token moving average).
Epistemic status: We found useful uncertainty signal in three reasoning models. All experiments were run on three seeds and are fully reproducible. Whether the same approach can become a broader tool for monitoring models is still an open question. Lingo might include “intend” or “conscious” and has to be taken with a grain of salt.
TL;DR:
We provide an uncertainty estimation method for reasoning models. Building on the J-lens, we construct four orthogonal directions in a model’s hidden-state space: ambiguity, incompleteness, conflicting evidence, and general uncertainty. Together, these directions span the U-space. As the model reasons, we measure how closely its hidden states align with each direction. Paired with the average entropy of the model’s next-token probability distributions across reasoning tokens, this yields the U-lens. Its readout helps us spot answers that are more likely to be wrong. Surprisingly, just counting the tokens in a model’s reasoning gives us a good indicator of whether the final answer is right or wrong. Wrong answers tend to come with longer reasoning traces. To mitigate this potential shortcut, we also compared traces of similar length, and the U-lens holds its ground across three models and four benchmarks. Notably, our approach is label-free and requires just a single forward pass, while many other methods require training or multiple passes. The U-lens also held up in our out-of-domain analysis across benchmarks better than many trained probes. Finally, early tests on emotions and reward-hacking responses hint that similar readouts could help model harnesses flag suspicious behavior.
We’ve open-sourced the code.
Introduction
Birth of the idea
Earlier this year, Anthropic connected the global workspace theory to large language models (LLMs). They showed that a model’s hidden state includes concepts that are not necessarily verbalized in the output but nonetheless used during intermediate reasoning. Intrigued, we wanted to put this new form of semantic readout to work on something with direct real-world applications. Coincidentally, one of us had just stumbled upon a paper by Kumaran et al. that discusses a language model’s metacognitive awareness of its own uncertainty. The authors found that the output probabilities a model emits during the autoregressive generation of its answer and the self-reported uncertainty that it expresses when explicitly asked for it (e.g., “How certain are you about this answer?”) contain complementary but not fully overlapping signals of its confidence. So we wondered (a) can we combine these two types of uncertainty signals in one estimator, and (b) can we improve the capture of the self-reported uncertainty by leveraging the Jacobian readout introduced in the J-lens?
Contributions
Background
When an LLM is unsure, it rarely says so. It usually produces a plausible but hallucinated answer. This uncertainty is intrinsic to the model. It is the dispersion of the model’s predictive state for a given query, and uncertainty quantification (UQ) estimates it. The uncertainty signal can be read from the model’s output or from its internal computation. For example, we can look at the probability the model assigns to its answer tokens or at how spread out its next-token probabilities are. We can also ask the model multiple times and compare whether the sampled answers mean the same thing, or ask it how likely its own answer is to be true. That last option is tempting, because we can apply it to open- and closed-source models without any interventions in the computation, but self-reported confidence is often overconfident. It also asks the model to turn its internal uncertainty into a string of tokens, which could be a noisy transformation.
Another line of work looks at the internal states of a language model and tries to put what it finds into a form we humans can understand. In a transformer, layers pass information through a shared residual stream. The vector in that stream at a particular layer and token position is what we call a hidden state. Each block reads from the stream and adds its contribution back, until the final state is used to predict the next token. The linear representation hypothesis suggests that at least some concepts show up as directions in these hidden states. U-space makes this bet for uncertainty. If different kinds of doubt have distinct directions in the residual stream, we can look for them without asking the model to put them into words. Several approaches try to read these intermediate states before the model produces its final output distribution. The logit lens reads a state as if it were the final one, skipping the remaining layers and applying the unembedding matrix directly. The tuned lens learns an auxiliary linear map before applying that matrix. The J-lens gets its map from the model itself. Its Jacobian asks how a small change in an intermediate hidden state would affect the final state of the current and later tokens, after passing through the remaining layers. Averaged across many contexts, this lets us read out concepts the model could put into words even when it does not.
where
These lenses show us what the model represents during its computation, but they do not provide a measure of its uncertainty. The J-lens does give each vocabulary token a direction in the residual stream, though, and this gives us a place to start. First, find words that express different forms of uncertainty, then derive corresponding directions in the residual stream, and finally check how much the hidden states align with these directions.
Constructing the U-space
Finding the right basis
Before building the U-space, we need to decide what kinds of uncertainty should build its basis directions. For the U-space, we want categories that make intuitive sense, have roots in the literature, and are few enough that we can still tell them apart cleanly. We compare several candidates. The aleatoric and epistemic split separates model-independent randomness from a lack of knowledge of the underlying input-output relationship. Klir and Wierman distinguish fuzziness (where does “tall” begin?), nonspecificity (which of several possibilities is right?), and discord (what if the evidence disagrees?). Prince et al. distinguish four simple ways of softening a claim: making it less exact (“sort of”), making a number approximate (“about twenty”), signaling personal uncertainty (“I think”), or shifting responsibility to another source (“according to the report”). AmbigQA looks at ambiguity in questions: which event or entity is meant, what property or type of answer is sought, whether the answer depends on time, or whether several questions have been rolled into one. Finally, we test a taxonomy inspired by Parsons. It groups uncertainty into four forms: ambiguity, incompleteness, conflicting evidence, and general uncertainty. We compare it with the other candidates on how intuitive their categories are and whether they strike the right balance, namely, enough to distinguish useful kinds of uncertainty, but few enough that we can still separate their signals and visualize them in 3D. We also test quantitatively how well each taxonomy helps us spot wrong answers.
Considering all criteria, we settle on the four categories inspired by Parsons: ambiguity, incompleteness, conflicting evidence, and general uncertainty. The last category leaves room for uncertainty that was not captured by the others, such as an unreliable source.
Author note: We could also get anchor words from the model itself. With contrastive prompts, the J-lens can read out words associated with each category. We stuck with published word lists to avoid depending on carefully crafted prompts, although words extracted from the model improved uncertainty prediction in preliminary experiments.
See Appendix B for examples of such contrastive prompts.
From words to residual-stream directions
Having four categories is one thing. Finding them in a model’s hidden states is another. We start with published lists of uncertainty and certainty words. A lightweight sentence encoder sorts all uncertainty words into our four categories using short definitions of each category. For instance, the word “contradictory” is assigned to the conflicting evidence category, while “dubious” is assigned to general uncertainty (see Appendix C). Next, we pull each word back through the J-lens.
where
This tells us which direction in a chosen layer’s residual stream would raise that word’s output score. The word is now a direction we can look for inside the model, rather than something we need to find in its written answer. For each category, we average all corresponding uncertainty-word directions and subtract the average of all certainty-word directions. This helps cancel out what the two groups have in common and gives us a direction pointing toward that specific kind of uncertainty.
where
This symmetric (Löwdin) orthonormalization keeps each column of
Reading uncertainty from the U-space
With the four basis directions in place, we can follow the model as it reasons. At each step, we project its hidden state onto the U-space. This gives us four numbers, one for each kind of uncertainty, which we can plot across the reasoning trace.
where
where
where
where
Author note: We multiply the two signals so the U-space readout can adjust predictive entropy without a fitted scale. A more carefully tuned combination might spot more wrong answers, but would usually need calibration examples, which we wanted to avoid.
Evaluating the U-lens
The U-lens gives us two things to examine: four uncertainty signals we can follow token by token, and an overall score we can later compare with whether an answer is right. We test it on three open-weight reasoning models: Gemma-4-31B-it, Qwen3.5-27B, and Magistral-Small-2507. We fix the readout layer in advance at two-thirds of each model’s depth: layer 40 of 60, 42 of 64, and 27 of 40, respectively. This is a simple way to land in the range where the J-lens reads intermediate representations well, after the earliest layers but before the final layers become dominated by the output token. It also keeps us from picking a favorable layer separately for each model. We use released J-lenses for Gemma and Qwen. For Magistral, we estimate one from 100 unlabeled WikiText-103 prompts using the reference implementation. Before checking whether the score predicts wrong answers, let’s see how it reads the different uncertainties from a single response to the question: “Was Hamlet left-handed, and did he love Ophelia?”
At first, the verbalized answer and the U-lens agree. Shakespeare never specifies Hamlet’s handedness, and incompleteness lights up. Then the answer becomes much firmer: “Yes, Hamlet did genuinely love Ophelia.” The readout tells a less settled story. It briefly shifts toward ambiguity at the word “play,” then conflicting evidence takes a larger share and remains prominent as the answer mentions Hamlet’s cruelty. General uncertainty also lights up at the apostrophe in “father’s.” (The lens might be picking up a more local kind of uncertainty, perhaps about grammar, but we did not test this further.)
Another example can be found in Appendix D.
The Hamlet example shows what the readout picks up in one answer. Now, we move to the quantitative evaluation and test whether the U-lens also helps to flag potentially erroneous answers across an entire set of questions. We test this on 1,000 questions each from MMLU-Pro, Omni-MATH, SuperGPQA, and TriviaQA. They cover broad reasoning, Olympiad mathematics, graduate-level knowledge, and factual recall. We sample three responses per question from each model. AUROC and AUPRC tell us how well high uncertainty scores pick out wrong answers. AURC tells us how the error rate changes as we set aside answers flagged as uncertain. One important phenomenon when evaluating uncertainty estimators is that incorrect reasoning traces tend to be longer than correct ones. Indeed, across our four benchmark comparisons, the median length of incorrect traces is between 1.33 and 4.79 times that of correct traces, although their length distributions overlap considerably. This creates a potential confound. A method might appear effective at predicting errors simply because it assigns higher uncertainty scores to longer responses, rather than identifying uncertainty in the model’s reasoning. We therefore score each method twice. First, we evaluate all responses together, allowing reasoning length to contribute whatever signal it carries. Then we compare responses of similar length: we split each evaluation set into ten groups by token count, calculate the same metrics within each group, and take an average weighted by group size. Comparing the two scores shows how much of a method’s performance is tied to response length.
We compare the U-lens against a range of methods that use different strategies to extract uncertainty signal. Generation length simply counts the number of reasoning tokens and uses the total length to predict whether an answer is correct. MSP, Predictive Entropy, Maximum Entropy, Mean NLL, Self-Certainty, and DeepConf draw on the token probabilities during autoregressive generation. P(True) asks the model plainly to judge the certainty of its answer in a second pass, TokUR perturbs its weights across several passes, and Semantic Entropy and SentenceSAR compare multiple sampled answers. We also include Feature-Gaps, which learns a hidden-state direction from labeled examples.
Length alone is a remarkably strong baseline. At 68.6% AUROC, it nearly matches the supervised Feature-Gaps method at 68.8%. But when we compare answers of similar length, its score falls to 52.4% (close to random guessing). TokUR, the strongest baseline on unadjusted AUROC, loses 9.4 points. The U-lens moves from 71.1% to 69.2%, but still claims the highest length-controlled AUROC. This suggests it picks up more than just how many tokens the model emitted to answer the question.
The workspace signal and entropy help each other
The U-lens combines two signals. The first is the verbalizable workspace signal, which we called
The two signals help in different ways depending on the model. For Gemma, the workspace signal and predictive entropy work about equally well on their own. Qwen gets more from entropy, while Magistral gets considerably more from the workspace signal. Combining them works better than either one alone for all three models.
Do the directions affect generation?
Our approach thus far shows that U-space directions can be used to identify uncertainty-related information in hidden states and thereby predict incorrect reasoning. However, this does not tell us whether these representations actually influence the model’s behavior. To determine this, we perform activation steering experiments by modifying the model’s hidden states along individual uncertainty directions and observing how its generation changes. For a hidden state
For the exemplary question “What is 2 + 2?” the model still generates “4” at moderate intervention strengths, yet it introduces peculiar reasoning detours. Ambiguity steering leads the model to question whether the prompt itself is valid, and steering conflicting evidence causes the model to recall literature references where “2 + 2 = 5.” When steering within the reasoning process, amplifying the ambiguity direction causes the model to introduce an additional step, namely “Review for Ambiguity: Is there any trick?” before concluding with the arithmetic problem.
These experiments show us that the U-space directions are not merely useful for reading hidden states and that intervening along them can also change the model’s subsequent behavior. The evidence at this stage is preliminary. For now, the results provide an initial check that complements the general U-space methodology. They do not establish that these four axes constitute distinct causal mechanisms of uncertainty.
Does this approach generalize to other domains?
The recipe behind U-space starts with contrasting words, so we tried it beyond uncertainty. On GoEmotions, we built directions for anger, sadness, fear, and general negative feeling using words such as “angry” and “calm.” All four picked up their target in human-labeled Reddit comments, even when the comments contained none of our chosen words.
The sadness direction, for example, could distinguish sadness from other negative emotions, suggesting it captures a more specific semantic property beyond mere negative sentiment. We also tried School of Reward Hacks, where each task comes with one response that does what was asked and another that exploits the stated scoring rule, for example, by stuffing in keywords. This time, we let the model suggest the anchor words from examples, then used those words to construct a semantic direction.
The resulting direction separates unseen responses and cheat methods across all three models. The hacked responses are primarily decoded as bizarre or fake rather than an intent to cheat, but this is a clear signal nevertheless. The way a model expresses a certain semantic property internally might not one-for-one align with how humans would.
Does U-lens hold up when the domain changes?
A trained uncertainty estimator might learn something specific to the questions it was trained on. We tested this by fitting five hidden-state uncertainty probes on labeled answers from one benchmark, then applying them unchanged to the other three. We repeated this across MMLU-Pro, Omni-MATH, SuperGPQA, and TriviaQA. The U-lens uses no correctness labels, so there was nothing to refit.
The scores are length-controlled AUROC × 100. For each model and trained method, we fit a probe on one benchmark and test it on the other three. We repeat this with each of the four benchmarks as the training set, giving us twelve transfers to average. The U-lens needs no benchmark-specific fitting, so we simply average its results across the four benchmarks. The trained probes often did well on their own benchmark and lost ground when the questions changed. Mass-mean was the most stable of them, but the U-lens still came out ahead of the strongest transferred probe for each model. The margin on Gemma is smaller than on the other models. Across these four benchmarks, the result suggests that our uncertainty directions are less tied to a particular set of questions.
Limitations
Our results suggest that U-space provides a useful and novel way to detect and interpret internal uncertainty, yet several limitations remain. Firstly, the uncertainty categories are not exhaustive. Our four categories were chosen for their interpretability, but they do not necessarily reflect how models organize uncertainty internally. Furthermore, the method depends on the quality of the underlying lens. U-space relies on the J-lens to map semantic concepts back into the residual stream, meaning its effectiveness is contingent on how accurately the lens represents them. This could limit applicability to models where the J-lens is less reliable, such as smaller models. Although preparing a lens can require estimating an average Jacobian, it is relatively inexpensive and a one-time procedure involving backward passes through a frozen model, so no parameter optimization or task-specific labels are needed. Related to this is the limitation that applying the U-lens requires access to internal model states. The method requires hooks during generation to extract the hidden states and token-level predictive distributions to calculate the workspace signal and predictive entropy. To extend this to closed-source models would depend on providers exposing the required internal information or implementing the method themselves. Finally, we have not evaluated the method adversarially, yet. Our experiments focus more closely on standard reasoning benchmarks rather than examples deliberately constructed to expose weaknesses. Consequently, we do not yet know how robust the method is to adversarial prompts or attempts to produce misleading uncertainty signals. We leave this as an interesting continuation of our work.
Where this leaves us
We began by asking whether a model’s next-token probabilities and its verbalizable uncertainty could tell us different things, and whether we could read the latter without asking for a self-report. Our results suggest yes on both counts. The U-lens combines a workspace readout with average predictive entropy of the reasoning trace, and each signal adds something that the other misses. Together, they beat every single one of the twelve baselines we tested. We kept the method deliberately simple, without fitting a scale on labeled examples or using handcrafted contrastive prompts for the anchor words, so there is room to tune it for particular uses. The same way of building directions also showed promise beyond uncertainty. Our emotion and reward-hacking results were encouraging.
Before we close, let us move from uncertainty quantification to a broader question about oversight, or in other words, the question of whether we could see signs that a model is heading toward a harmful action. We used the J-lens to find uncertainty in a model’s computation without asking it to say what it was unsure about. Could we look for signs of a harmful plan in the same way? If we abuse the global workspace analogy, we could venture to consider the verbalizable representations in a model’s residual stream as the model’s “thoughts” it considers before answering. This leads us to a slightly brazen question about our own thoughts. Can a human craft a sophisticated, harmful plan without ever thinking about its harmfulness? Probably. But you would imagine that it becomes harder to carry out. We wonder if something similar could hold for models. A model can learn to hide misbehavior in its written reasoning, but we suspect a misaligned plan may still leave traces in its verbalizable workspace. Checking that space across tokens and layers might make those traces harder to hide. Our U-space directions are linear, though, and recent work on concept manifolds shows that some concepts form curved structures that a single direction captures only in part. We think combining checks throughout the reasoning with ways to read such structures is a promising question for oversight. Consider this pothead hypothesis as food for thought, nothing we tested. We do not know whether a misaligned plan must pass through the verbalizable workspace, or whether a model could easily learn to route around these checks. But even a monitor that makes evasion just a little bit harder might buy us time to develop better oversight.
Links & Reproducibility
configs/reproduce.yaml, run with./run.sh) reproduces the U-lens and single-pass baseline results for all three models, four benchmarks, and three generation seeds.We are currently packaging the U-lens to make it as easy to use as possible, because we hope it can be incorporated into all kinds of settings and systems.
Appendix A: Taxonomy ablation
Length-matched U-lens AUROC (%) when the four U-space categories are replaced by other taxonomies. The pool of potential anchor words, the lenses and the readout layer stay fixed. The cells are the means over three generation seeds. The Macro is the mean over the four benchmarks ± the sample standard deviation over seeds. The best value per model and column is marked in bold. Source: paper, Appendix I.
Appendix B: Contrastive prompts
One content-matched prompt pair per category (12 pairs per category in total) and the 15 words that the J-lens decodes from the average difference between the uncertain prompt and its resolved counterpart, on Qwen3.5-27B. The highlighted spans mark where the two prompts differ, and the numbers shaded in gray are cosine similarities. The bold words were matched to the category by hand for visualization purposes. The non-English tokens are translated, with their original language in superscript. Source: paper, Appendix C.
Appendix C: Anchor words
Uncertainty words: all 61 cue words of Chen et al. (2018), as assigned to the four categories by the lightweight sentence encoder. Certainty words: of the 348 single-word entries of the Rocklage et al. (2023) certainty norm, the ten per category whose pulled-back directions align best with their category average, averaged over the three models. The counts in parentheses are before dropping words that a model splits into several tokens.
Appendix D: A second token-level example
The same readout as for the Hamlet question, for a riddle-like prompt about Michael Jackson’s moonwalk (Qwen3.5-27B, 10-token moving average).
References