This is a special post for quick takes by _rpd. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
we present BonaFide, a benchmark of 3,066 labeled CoTs across 13 tasks and 10 models, and use it to conduct the first systematic evaluation of prominent faithfulness metrics. Our experiments show that most metrics perform near chance, exhibit strong prediction biases and degrade on longer CoTs.