x

Lucas Florin

Subscribe

Message

PhD Student in the AI Safety group at the University of Stuttgart.

31

1

6mo

Lucas Florin

Subscribe

Message

PhD Student in the AI Safety group at the University of Stuttgart.

31

1

6mo

LLMs Knowingly Hide Their Mistakes

TL;DR: We prefill models with synthetic mistakes and then send a follow-up message. * Models fail to disclose the mistake in 67% of cases. * In 5% of cases, they notice the mistake in their chain-of-thought (CoT) but don't disclose it anyway. This is deception by omission. * This means...

Oct 1316

Verbalised evaluation awareness in language models has little effect on their behaviour

by Amelie Knecht, Lucas Florin, and Thilo Hagendorff

TL;DR: We provide evidence that the presence of verbalised evaluation awareness (VEA) in CoTs does not imply eval gaming. We tested this across 8 open-weight LRMs and 4 benchmarks (safety, alignment, moral dilemmas, political opinion) by comparing answer distributions on the same prompts, with and without VEA in the CoT....

May 1219