Thanks to Jonathan Richens and Tom Everitt for discussions about this post.

The case for causality research^[1] is under-stated in the broader AI Safety community. There are few convincing arguments floating around for why and how research in causality can help us understand and mitigate risks from AI.

Cause and effect relationships play a central role in how we (and AIs) understand the world and act upon it. Causal associations intend to be more than descriptions, or summaries of the observed data, and instead relate to the underlying data-generating processes. The overall pitch is that such a mechanistic understanding, when recognised and mastered, allows one to infer what would happen under interventions and hypothetical (counter-to-fact) scenarios, that can give us a richer understanding on AI behaviour and risks.

Here I compile a list of different arguments I have collected from people currently pushing this agenda as well as my own research for why causality matters.

Theories of impact

1. Understand the limits of AI capabilities. There are limits to the things we can observe and experiment on. It is unlikely that recorded data will ever capture the outcomes of all possible experiments (including counterfactual thought experiments) under all circumstances. The implication is that answers to these questions must be deduced from other data and assumptions about the world. For example, we might want to know whether an AI agent could eventually, with enough training, predict the effect of any experiment in some domain, even those that haven’t yet been conducted. The answer will sometimes be yes and sometimes no. How to reach these conclusions can be interpreted as a problem of causal identification (e.g. Def. 17 in Bareinboim et al. 2022).

2. Provide a foundation for AI world modelling. Causal models are immensely expressive, encoding the likelihood of events and the behaviour of the system under (possibly incompatible) interventions. Inferring the AI’s world (or causal) model from the data and assumptions available is a problem of causal discovery (Glymour et al., 2019). Many different methods have been developed under different assumptions and could be leveraged to study the models underlying AI behaviour.

There is already lots of convincing theoretical evidence that AI systems that behave approximately rationally (Halpern and Piermont, 2024) or that ar...

Posts

Wikitag Contributions

Comments