LESSWRONG
LW

likenneth

Message

https://likenneth.github.io/

194

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

Excited to announce our new work: Inference-Time Intervention (ITI), a minimally-invasive control technique that significantly improves LLM truthfulness using little resources, benchmarked on the TruthfulQA dataset. Preprint link. We start from the surprising finding that certain attention heads have a clear activation distribution difference for true and false statements. Probing...

Jun 11, 2023•195

likenneth

https://likenneth.github.io/

likenneth — LessWrong

likenneth

Message

https://likenneth.github.io/

194

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

Jun 11, 2023•195

likenneth

https://likenneth.github.io/

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

likenneth

We start from the surprising finding that certain attention heads have a clear activation distribution difference for true and false statements. Probing at these points yields upwards of 83% accuracy on TruthfulQA while zero-shot generation is only at 30% accuracy.

We propose Inference-Time Intervention (ITI): shifting the activations along the difference of the two distribution means during inference time; model weights are kept intact. The same intervention process is repeated for generation of each token autoregressively.

Here’s an example. For the same user question, LLaMA with (blue)... (read more)

195