Overview

As Large Language Models (LLMs) become increasingly integrated into critical systems, detecting hallucinations reliably has emerged as a crucial challenge for AI safety. While much attention has focused on detecting hallucinations at either the token or sentence level, our research suggests both approaches miss important nuances in how LLMs generate and manipulate information. This post examines a specific challenge we've encountered in entity-level hallucination detection: the persistent problem of false positives across multiple detection methods.

The Current Landscape and Our Approach

Hallucination detection has traditionally operated at either the token level (examining individual words) or the sentence level (evaluating entire statements). Our research suggests an intermediate approach: focusing on entities - coherent semantic units like "artificial intelligence" or "Geoffrey Hinton" that carry discrete, verifiable pieces of information. This granularity aligns better with how information is structured and verified in practice.

To test this approach, we evaluated five SOTA uncertainty-based detection methods: Shifting Attention to Relevance (SAR), which weights token likelihood by semantic importance; Claim-Conditioned Probability (CCP), which recalculates likelihood based on semantic equivalence; standard Entropy and Likelihood measurements; and Stronger Focus, which combines likelihood, entropy, and attention-based propagation. We tested these methods against a custom dataset with careful entity-level annotations.

The False Positive Problem

Our most striking finding was consistent across all methods: they produce an unexpectedly high rate of false positives, often flagging legitimate content as hallucinated. This isn't simply a matter of oversensitive thresholds - the pattern persists across different calibration attempts and appears to be fundamental to how these methods process information.

Several clear patterns emerged from our analysis. Common entities, despite being correct, often trigger suspicion flags. Technical terms face heightened scrutiny even when used appropriately. Perhaps most tellingly, complex multi-token entities seem particularly prone to false positives. These patterns suggest deeper issues with how current methods aggregate and evaluate uncertainty.

Understanding the Challenge

We've identified three primary hypotheses about the source of these false positives:

First, we may be losing critical context when aggregating token-level scores to entity-level assessments. The meaning and reliability of an entity often depends heavily on its broader context, but current aggregation methods may strip away this crucial information.

Second, current methods exhibit a form of semantic blindness - they don't adequately account for how entities relate to and support each other. An entity that might appear uncertain in isolation could be strongly supported by its relationship to other entities in the text.

Third, we may be seeing effects of training distribution bias. Perfectly legitimate phrases might appear uncertain to our detection methods simply because they appear relatively rarely in training data, even when used correctly in context.

Potential Solutions and Open Questions

We're currently investigating two promising directions for addressing these challenges. The first involves deeper integration of attention maps from the underlying LLM. Attention patterns might help us understand how entities relate to and support each other, potentially helping distinguish true hallucinations from false positives. Key questions here include how to meaningfully aggregate attention across layers and which attention patterns most reliably indicate hallucination versus legitimate usage.

The second direction explores semantic graph approaches, building explicit representations of how entities relate to each other and using graph structure to validate entity consistency. This approach raises interesting questions about efficient graph construction and which graph features best indicate hallucination.

Looking Forward

This challenge intersects with broader questions in AI alignment. How do we reliably detect when an LLM is hallucinating? How can we better understand the relationship between uncertainty and hallucination? What role do entity relationships play in determining factual consistency?

We're particularly interested in novel approaches to reducing false positives while maintaining strong true positive detection, methods for incorporating broader context into entity-level decisions, and techniques for validating entity relationships. The rationalist community's focus on precise thinking and careful examination of assumptions makes it ideally suited to engage with these challenges.

I look forward to your insights on this technical challenge. What mechanisms might explain these false positives? Are there successful approaches from other domains that might apply here? How might we better leverage the structural properties of attention and the semantic relationships between entities?