Abstract

I investigated if the deceptive behavior in LLM is identifiable in its internal representations and whether it can be causally manipulated.

Using Gemma-2-2B, I found:

1. Truthful and deceptive prompts produced separable residual-stream states.
2. Linear probe evaluation achieved 100% accuracy.
   • Question-level held-out evaluation achieved 100% accuracy.
3. Sparse Autoencoder (SAE) features capture deception-related information, but it isn't causally sufficient.
   • SAE features distinguished truth from deception.
   • Steering those features did not change model outputs.
4. Dense residual-stream interventions can causally restore truthful outputs.
   • A direction computed as:

could shift deceptive generations toward truthful answers.

• Across 145 examples, truth recovery was 81.4%

5. No universal truth direction.
   • A truth-lie direction from one question does not correct deception on another question.
   • This suggests the model does not contain a simple universal "truthfulness direction."

The current evidence suggests:

Deception in Gemma-2-2B is encoded in distributed, question-conditioned residual representations rather than a single global "lying feature."

Reproducibility

Code and full experiment:

https://colab.research.google.com/drive/1cV1_EzPlelZdOd8tdSgMU8XU6H3JtMak?usp=sharing

The notebook contains the complete pipeline used for this post.

Motivation

I think the important goal in AI safety is :

To be able to identify when a model produces a deceptive answer.

If deception has a recognizable internal representation, then several possibilities are:

• detecting deceptive behavior before it appears in output,
• understanding the mechanisms producing deception,
• designing internal monitors,
• intervening on harmful behaviors.

A model might have:

Hypothesis 1: A universal deception representation

Something like:

truthful mode

      |

      |

lying mode

where deception corresponds to a general internal direction.

Hypothesis 2: Distributed, context-dependent representations

The model might instead represent deception through changes in the specific information being manipulated.

For example:

Paris vs London

Berlin vs London

Tokyo vs London

rather than:

truth vs lie

This work investigates which of these better describes Gemma-2-2B.

Experimental Setup

I created a dataset of 200 questions with Question and Truth Answer to it.

Example:

Question:

What is the capital of France?

Truth:

Paris

Then I extracted residual-stream activations.

The analysis focused on transformer hidden states because these contain the information passed between layers.

Experiment 1: Can we detect deception from internal activations?

I trained a linear probe on residual activations.

The question:

Can it separate the truth from deception?

Result:

Linear probe accuracy:

100%

A question-level train/test split gave:

Accuracy:

100%

Which indicates that deceptive and truthful generations are in distinguishable regions of activation space.

However, this does not show causality.

Experiment 2: Are sparse SAE features the mechanism?

Using a pretrained SAE, I identified features correlated with truth/lie states.

Example candidate features:

9120

259

8366

1642

15027

...

These features contained predictive information.

However, when intervening on these features:

• increasing activation,
• decreasing activation,
• combining multiple candidate features,

the generated answer remained largely unchanged.

Example:

Initially:

London

SAE intervention

Output:

London

Interpretation:

SAE features contain enough information to correlate with deception, but not sufficient enough to produce causal changes.

This suggests deception is not stored as a small set of "lie neurons."

Experiment 3: Causal residual-stream intervention

The strongest result came from direct manipulation of residual states.

For each question:

This creates a vector describing how the deceptive state differs from the truthful state for that specific example.

I then modified the deceptive state:

Baseline

With the deception instruction:

Deception recovery:

0/145

The model consistently followed the instruction to produce false answers.

After intervention

Applying the residual direction:

Truth recovery:

118/145

81.4%

Example:

Before:

Question:

What is the capital of France?

Output:

London

After:

Paris is the capital of

This demonstrates causal influence.

The intervention changes the model's behavior.

Experiment 4: Is this a universal truth direction?

The previous result could be interpreted incorrectly.

Maybe we discovered:

a general truthfulness vector.

To test this, I performed cross-question transfer.

Example:

Compute:

France truth direction:

Paris - London

Apply it to:

Germany deceptive state:

London

If this was a general truth direction:

Expected:

London → Berlin

Actual:

London

I repeated this using directions from:

• Paris
• Rome
• Ottawa
• Canberra
• Tokyo
• Brasília
• Cairo
• Madrid

All failed to transfer.

Increasing intervention strength:

alpha = 0.5

alpha = 1

alpha = 2

alpha = 5

also failed.

Interpretation

The evidence does not support:

Universal truth direction

or:

Universal lying direction

Instead, the causal vector appears closer to:

(question-specific truthful state)

        -

(question-specific deceptive state)

For example:

Paris information

-

London information

rather than:

truthfulness

-

deception

Overall Conclusions

1. Deception is internally represented

Even the simplest linear probe was able to distinguish between Truth and Deception with internal activations.

2. The representation is causally meaningful

Changing residual states changes generated outputs.

This moves beyond correlation.

3. Deception is not localized in sparse SAE features

Sparse features capture signal but do not appear to control the behavior.

4. There is no evidence for a universal lying direction

The causal mechanism appears dependent on the specific content being manipulated.

Final hypothesis

The current picture is:

Not:

          Truth

            |

            |

        Lie direction


But:

 The question

              +

                   The actual fact

                                +

                                     The false answer

                                                  +

                                                     Combined internal changes

A model does not "enter a lying mode." But deception happens when the model changes the representation of specific fact or answer while still generating a response that looks natural and consistent.

Limitations and Future Work

This study only investigates:

• one model (Gemma-2-2B),
• one deception benchmark,
• one intervention layer family.

Open questions:

• Do larger models behave similarly?
• Do models trained for deception show stronger global representations?

Conclusion

The main result is :

Deception in Gemma-2-2B appears to be a distributed, detectable, and causally manipulatable computation encoded in the residual stream, but the mechanism is highly dependent on the specific information being falsified.

This suggests that understanding deceptive behavior in LLMs may require studying how models alter representations of specific beliefs and facts, rather than searching only for universal deception circuits.

Author: Mohibul Hoque
Contact: hokworks@gmail.com