The Alignment Project is a global fund of over £15 million, dedicated to accelerating progress in AI control and alignment research. It is backed by an international coalition of governments, industry, venture capital and philanthropic funders. This post is part of a sequence on research areas that we are excited to fund through the The Alignment Project.

Apply now to join researchers worldwide in advancing AI safety.

Economic Theory and Game Theory

We are excited about using framings and tools from theoretical economics to better describe incentives and equilibrium outcomes relevant to AI alignment. By considering AI agents in a strategic setting, we hope to ensure alignment approaches are robust to capabilities improvements and that these framings can inspire new ideas for alignment techniques. We expect useful insights to come from proposing new or different abstractions: how different forms of misaligned incentives impact outcomes or how robust proposals are to different equilibrium concepts or modelling assumptions.

Hammond et al., 2025 provide a taxonomy of risks from advanced AI in multi-agent settings, including a broader list of problems and further context on some of the specific problems we highlight below.

Information Design

Problem summary: Current frontier models already exhibit sycophantic and deceptive behaviour (OpenAI, 2025; Park et al., 2023). This is arguably evidence of strategic behaviour. As AI models become more strategic, how should we expect models to choose what information to reveal? We anticipate useful insights from areas within information design (Bergemann and Morris, 2019) such as cheap talk (Crawford and Sobel, 1982), verifiable information disclosure (Milgrom, 1981), persuasion (Kamenica and Gentzkow, 2011) or delegation (Holmstrom, 1984). What are optimal revelation strategies from the perspective of the AI, how varied and stable are equilibria, and can we design mechanisms to improve information disclosure?

Why this matters: Information revelation matters for several aspects of alignment. Some alignment proposals focus on building protocols to incentivise honesty (e.g. Irving et al., 2018). If we trust an AI to be honest or incapable of deceiving us, we may be able to reliably use it to supervise or design other models (e.g. Buhl et al., 2025). In addition, information design may be able to help determine when we can use mi...

Posts

Wikitag Contributions

Comments