TL;DR: We examine whether LLMs can explain their own behaviour with counterfactuals, i.e. can they give you scenarios under which they would have acted differently. We find they can't reliably provide high-quality counterfactuals. This is (i) a weird and interesting failure mode, and (ii) should be a concern for high-stakes deployment. When a model gives a misleading explanation of its behaviour, users build incorrect mental models, which can lead to misplaced trust in systems and incorrect downstream decision-making.

📝 arXiv paper
📊 EMNLP poster
⚙️ GitHub

The importance of self-explanations

In an ideal world, we'd evaluate models in every domain possible to confirm that their predictions are not based on undesirable variables, their reasoning steps are consistent with our prior beliefs, and their intentions are aligned with our goals. Of course, this sort of evaluation is impossible for general systems. Instead, we demand a proxy: explainability. Being able to explain decision-making means we can evaluate these properties on the fly. Explainability is a way of facilitating trust that models will behave as intended on unseen prompts, without requiring a comprehensive audit.^[1]

There are many paradigms of explainability/interpretability, but LLMs offer a new one: self-explanation. Perhaps we can just ask the model to explain its own behaviour? Natural language explanations are easy to generate, more flexible than traditional techniques, and preferred by users.

However, if self-explanations are unfaithful to the actual underlying reasoning, they can mislead. You might conclude a model is not making decisions based on a certain undesirable variable when, in fact, it is. Or you might conclude the model isn't hacking the coding test when, in fact, it is.

Self-generated counterfactual explanations

One form of self-explanation are self-generated counterfactual explanations (SCE). These are where a model explains its prediction by providing a counterfactual that would have caused it to predict a different, specific outcome.

For example, say a model is deployed in a healthcare setting. It might predict that a 60-year-old male with a systolic blood pressure of 135 mmHg is at high risk of developing heart disease. In response, a clinician might ask: "What would need to be different for the model to predict low risk instead?" The model could respond with a counterfa...

Posts

Wikitag Contributions

Comments