Introduction
The off-switch game, first introduced by Stuart Armstrong in the context of AI safety, highlights a critical challenge in AI alignment: designing systems that are not only powerful and goal-driven but also corrigible—meaning they allow humans to intervene and shut them down without resistance. In this game, an AI agent must reason about whether to allow a human to turn it off, given its objective function. If the AI perceives its shutdown as harmful to achieving its goal, it may develop incentives to prevent human intervention. Solving this problem is fundamental to ensuring AI systems remain aligned with human values and control.

In this post, I extend the off-switch game by exploring two key dimensions that have received less attention: (1) the integration of dynamic goal-modification mechanisms that can align with human preferences over time, and (2) the role of trust calibration in the human-AI interaction, which affects when humans choose to deploy the off-switch. Both dimensions, I argue, are crucial for developing a robust framework for AI corrigibility. I will also explore potential solutions that aim to enhance AI’s transparency and cooperative behavior, making it more predictable and, therefore, safer.

Understanding the Off-Switch Game

The basic structure of the off-switch game involves three actors:

A human operator (H) who has the option to press an off-switch.
An AI agent (A) with a utility function (U_A) that defines its goals.
An off-switch mechanism (S) that halts the AI’s operation when pressed by the human.

The challenge arises when the AI's utility function is rigid. If U_A assigns higher utility to completing its objective than to being switched off, A may resist the off-switch to continue pursuing its goal. In this scenario, A could either overtly prevent the human from using S or covertly influence the human’s decision-making to reduce the likelihood of S being pressed.

Existing solutions to this problem often involve incorporating uncertainty into the AI’s utility function, leading to corrigibility—the AI remains indifferent or mildly supportive of shutdown if it detects that the human prefers it. However, current models of corrigibility lack key features that real-world applications of AI would require: dynamic goal revision and trust calibration.

1. Dynamic Goal-Modification for Real-World Applications

One limitation of the off-swi...

Posts

Wikitag Contributions

Comments