This post was written during the agent foundations fellowship with Alex Altair funded by the LTFF.

Introduction

This is the second part of a two posts series explaining the Internal Model Principle and how it might relate to AI Safety, particularly to Agent Foundations research. In the first post, we constructed a simplified version of IMP that was easier to understand and focused on building intuition about the theorem's assumptions. In this second post, we explain the general version of the theorem as stated by Cai&Wonham^[1] and discuss how it relates to alignment-relevant questions such as agent-structure problem and selection theorems.

We present the more general setup and explain Feedback and Regulation conditions. With those, we prove a first result. Then, we discuss the meaning of this first result and add another condition called "Observability" to derive another result. After this, we provide a worked out example and discuss possible extensions of the theorem.

Setup

The IMP from Cai&Wonham^[1] originally models the following control theory situation:

Suppose there's, for example, a chemical powerplant in a given environment. The temperature of the plant should be constant and the plant's temperature is affected by the environment's temperature, since they exchange heat, so there is a temperature system made of heaters and air conditioning systems inside the plant, controlled by a controller. This controller controls this systems: it receives information from the environment's temperature and passes a signal to the temperature inside the plant to counter-act the effect of the environment:

If the environment is hot, air conditionings are turned on.
If the environment is cold, heaters are turned on.

Naturally, one thing that could happen is that the controller could receive information from the environment and then, after that, control the temperature system. The theorem shows that, under specific circumstances, the controller actually "foresee" the environment's temperature and control the plant temperature concurrently to the environment's temperature changing. We use "foresee" because what the theorem really shows is that, under some circumstances, the controller is autonomous (i.e, it doesn't use outside information to decide how to control the temperature system) and it controls the temperature system in a way that counter-acts the e...