I just became aware of the good regulator theorem through John Wentworth's post. Then I tried to read the paper he talked about, Every Good Regulator Of A System Must Be A Model Of That System. This is among the worst papers I've ever skimmed. It's barely comprehensible. Reading this paper is not like reading a regular text. I had to try to read their intentions instead.

This post is my attempt to make sense of the setup and the results of the paper. Summed up, my conclusions are:

  • Their main theorem is correct but trivial and not very interesting. (It can be strengthened too, see the proposition below.) Their proof is a mess I haven't bothered looking at, it's way too complex.
  • Their interesting conclusion, that an optimal action can only be made if you have a model of the system (i.e., everything outside your control), is false.
  • Some thoughts about terminology:
    • It's definition of "most simple" probably doesn't make sense.
    • The definition of "isomorphisms" at least makes sense, but shouldn't have been called "isomorphism".

Setup of the paper

I don't follow the notation or terminology in the paper as it can be replaced with more familiar notation from decision theory.

Define the sets (the name in the paper in parenthesis) Define the functions For each uncontrollable outcome we have a non-empty set describing the good outcomes. Two examples are:

  • Goal sets. Let be a goal set, a set of outcomes that are good, independently of the value of . Assume that for all there is an action so that . Then is non-empty for all . This is what they do in the beginning of their paper.
  • Utility maximization. Let be a utility function and assume that exists for all . Define . In the paper they talk about minimizing entropy specifically, i.e., .

Note: The authors don't use ; they only use the entropy.

Our goal is to find an optimal action function that satisfies In other words, we want to make sure that every outcome is good. This is the regulation problem.

The regulation problem is equivalent to a utility-maximization problem with no randomness. Let be an arbitrary utility function. An optimal action according to the regulator problem satisfies

but that is equivalent to the optimal action as defined by ordinary decision theory, The other way around is similar.

The theorem

The point of the paper is to prove that any simple action function has to be isomorphic to the mapping . Their main theorem states

Theorem: The simplest optimal regulator of a reguland produces events which are related to the events by a mapping .

In my notation this is

Theorem: The "simplest" optimal action function can be written as for some function .

In the abstract they claim that "making a model is necessary," where "making a model" presumably means that you have to reconstruct in order to calculate the optimal action . This conclusion is false. I guess they made the mistake due to their misapplication of the word isomorphism.

Their notion of isomorphism (used elsewhere in the paper, including in the abstract, but not in the main theorem) is quite straight-forward, but not similar to any other use of word that I've seen. They say (equation 8) that two functions are isomorphic to one another if there is any so that . This is not even close to the spirit of isomorphism, as the fundamental property of isomorphisms is to preserve structure perfectly. For instance, isomorphisms should be transitive, i.e., if and then . This is obviously not the case in this example, as it's easy to see that the constant function is "isomorphic" to any function. I could agree to and being homomorphic, as homomorphism do not require invertibility, but isomorphism is just simply the wrong word. (The point here is not to argue over definitions, just that you shouldn't use words in a way that confuses yourself and other people.)

That said, their notion of isomorphism isn't useless. It allows you to calculate the value of from the value of . And that is a nice property. But it doesn't tell you that you need to calculate , as claimed in the abstract. If we were talking about real isomorphisms on the other hand, you would've always been able to at least recover from by the inverse mapping -- and I think that's closer to what they actually wanted to show.

I haven't made much of an effort to understand their usage of the word "simple". But it turns out it doesn't matter, because their result holds for all action functions , not only the "simple" ones.

What is the spirit of their claims? I think the point is that you should expect to be either

  • (a) directly used in the computation of , or
  • (b) recoverable from , at least provided we know .

Both claims are really easy to disprove, you only need to have independent of the outcome . Then we don't need to use and we can't recover from if the space contains more than one element. I care about claim (b) because it's one of the few ways to formalize that "uses" or "knows" . Recall that is just a function, a collection of ordered pairs . This function doesn't know anything. (I think this might be relevant for the ELK problem too. A function doesn't have any latent knowledge. It's just a function.)

The following proposition is, as far as I understand their paper, an extension of their main theorem. There they claim there is a mapping that produces the optimal outcome in the sense that . This is true, and in fact an even stronger statement is true. But it doesn't mean what they think it means.

Proposition

Let be a parameterized family of choice functions on . Define by Then

  • (a) The function is an optimal action function (a solution to the regulation problem),
  • (b) If is an optimal action function (solution to the regulation problem) there is a family of choice functions so that ,
  • (c) Even if we know and one of the families of choice functions so that , we will not always be able to reconstruct .

Proof

The first two claims are trivial. The third was explained above. The point of the parameterized choice functions is to make it possible to construct functions that choose any element from . There are probably more elegant ways to formulate it.

Rough explanation

The regulation problem is equivalent to a utility-maximization problem as already explained. A function is an optimal action function iff it satisfies Here can be a set, that's why I use the symbol. There are potentially many optimal action functions. For instance, if and each set contains two elements, there are action functions. This should explain the rôle of the choice functions. You can see from the defining property of that it only depends on through ; this explains the existence of the functions.

New to LessWrong?

New Comment