This is an alternative version of UDT 1.0 tiling theorem described in Understanding Trust paper. Work was done during 2025 AI Safety Camp under mentorship of Abram Demski.
Notation
Let denote the space of observations and denote the space of actions, with being the specific actions available when facing observation . Let denote the set of all policies which map observations to actions . Note that any persistent internal state (i.e., memory) is treated as part of the observation. The actual policy chosen by the agent will be denoted by .
A policy-point is a pair . I will treat a policy as a set of policy-points, so that is synonymous with .
We distinguish a subset of self-modifying actions. (Note that a self-modifying action modifies the agent's policy, and not the action itself.) The set of non-self-modifying actions is denoted as . For any , let denote a non-self-modifying version of , which appears identical from the outside perspective.
For a self-modifying action , let denote the set of policy-points modified by , let be the set of actions which the self-modifying action writes in the policy-points, and let be the set of observations in the modified policy-points. We also enumerate the actions, such that for all , , it is true that .
(Really, I imagine agents as having source code defining their behavior, and self-modifying actions would edit the source code rather than directly modifying the policy. However, it is simpler to deal with the policy-point modifications directly. A more realistic treatment should recognize that the list of policy-points modified is derived from a more fundamental understanding of the consequences of a self-modifying action.)
For the generalized proof, we also need to introduce a hierarchy of self-modifying actions. We define the order of an action, , to be:
UDT 1.0 Tiling
Here are some modified assumptions. (The approach was inspired by work done by Linda Linsefors and Alex Mennen during an internship at the Machine Intelligence Research Institute.)
Assumption 1 (Limited Self-Modification). Any self-modifying action modifies exactly one policy-point , where .
This assumption limits self-modifying actions to only change one policy-point, and more importantly, not to force other self-modifying actions. This avoids chains of self-modifications which propagate other self-modifications.
Assumption 2 (Fine-Grained Fairness). For any self-modifying action :
This assumption says that the expected utility of taking a self-modifying action is equal to the expected utility of the corresponding non-self-modifying action plus the knowledge that the policy already takes the action which the self-modification would have forced.
Assumption 3 (Faith in Argmax).
This assumption says that the agent expects argmax to achieve (at least) the maximal value (not only for the basic case of selecting a single action, but also in the presence of further information ).
Assumption 4 (Action Coordination). For any observations and actions :
What this assumption says is that the agent doesn't expect knowledge of a different policy-point to change the optimal decision with respect to .
Assumption 5 (Knowledge of Decision Procedure).
Theorem 1. Assuming Fine-Grained Fairness, Limited Self-Modification, Faith in Argmax, Action Coordination, and Knowledge of Decision Procedure, UDT 1.0 does not strictly prefer any self-modifying action.
Proof. Suppose for contradiction that some self-modification is strictly preferred:
By Fine-Grained Fairness, the expectation of the self-modifying action is equal to the expectation of the corresponding non-self-modifying action , when conditioned on knowledge that the forced action would be taken anyway:
By definition of max, this expectation is at most the expectation of the best action which could be substituted for :
By Faith in Argmax, this is at most the expected utility conditioning on the abstract statement that the best action will be chosen, rather than conditioning on the concrete best action:
By Action Coordination, I can drop one of the conditions inside the argmax:
By Knowledge of Decision Procedure, I can drop the argmax condition entirely:
Putting it all together, the self-modifying action is just as good as its non-self-modifying version:
This contradicts the initial assumption.
The generalized version
We can also have a generalized version of this proof. For this, we need to change some assumptions. Instead of Limited Self-Modification we have:
Assumption 6 (Hierarchical Self-Modification). For any self-modifying action :
i.e. the order of the self-modifying action is well-defined, and
Assumption 7 (Finite Self-Modification). For any self-modifying action :
Those assumptions ensure that all self-modifying actions eventually "roll out" (because the order is finite and the number of policy-points which are modified is finite) into non-self-modifying ones, without loops.
Instead of Fine-Grained Fairness, we have:
Assumption 8 (Faith in Fairness). For any self-modifying action , and for any statement of the form with :
This is just as Fine-Grained Fairness, but we added that an action can modify multiple policy-points, and that conditioning on setting other policy-points does not interfere with the expectation equality.
We also need to generalize action coordination:
Assumption 9 (Generalized Action Coordination). For any observations and actions :
What this assumption says is that the agent doesn't expect knowledge of different policy-points to change the optimal decision with respect to .
Theorem 2. Assuming Faith in Fairness, Hierarchical Self-Modification, Finite Self-Modification, Faith in Argmax, Generalized Action Coordination, and Knowledge of Decision Procedure, UDT 1.0 does not strictly prefer any self-modifying action.
Proof. Suppose for contradiction that some self-modification is strictly preferred:
By Hierarchical Self-Modification, this action has a finite order .
By Faith in Fairness, the expectation of the self-modifying action is equal to the expectation of the corresponding non-self-modifying action , when conditioned on knowledge that the forced actions would be taken anyway:
The actions all have order at most (by the definition of order).
Applying Faith in Fairness to each of them (which is possible, since all other statements in the brackets are of the form of a conjunction of conditions on policy-points), the maximum order of actions in the policy-points being conditioned on decreases to . I can repeat this until all of the actions in the brackets are of order . Since is finite, and on each step I add only a finite number of policy-points (by Finite Self-Modification), the final statement also contains a finite number of policy-points.
Then I get an expectation of the following form:
where , and . For each we define a statement .
By definition of max, this expectation is at most the expectation of the best action which could be substituted for :
By Faith in Argmax, this is at most the expected utility conditioning on the abstract statement that the best action will be chosen, rather than conditioning on the concrete best action:
By Generalized Action Coordination, I can drop the policy-point conditions inside the argmax:
By Knowledge of Decision Procedure, I can drop the argmax condition entirely:
The index of the statement dropped by . Doing this procedure times, I can drop the statement entirely (bringing the index to ):
Putting it all together, the self-modifying action is just as good as its non-self-modifying version:
This is an alternative version of UDT 1.0 tiling theorem described in Understanding Trust paper. Work was done during 2025 AI Safety Camp under mentorship of Abram Demski.
Notation
Let
A policy-point is a pair
We distinguish a subset
For a self-modifying action
(Really, I imagine agents as having source code defining their behavior, and self-modifying actions would edit the source code rather than directly modifying the policy. However, it is simpler to deal with the policy-point modifications directly. A more realistic treatment should recognize that the list of policy-points modified is derived from a more fundamental understanding of the consequences of a self-modifying action.)
For the generalized proof, we also need to introduce a hierarchy of self-modifying actions. We define the order of an action,
UDT 1.0 Tiling
Here are some modified assumptions. (The approach was inspired by work done by Linda Linsefors and Alex Mennen during an internship at the Machine Intelligence Research Institute.)
Assumption 1 (Limited Self-Modification). Any self-modifying action
This assumption limits self-modifying actions to only change one policy-point, and more importantly, not to force other self-modifying actions. This avoids chains of self-modifications which propagate other self-modifications.
Assumption 2 (Fine-Grained Fairness). For any self-modifying action
This assumption says that the expected utility of taking a self-modifying action
Assumption 3 (Faith in Argmax).
This assumption says that the agent expects argmax to achieve (at least) the maximal value (not only for the basic case of selecting a single action, but also in the presence of further information
Assumption 4 (Action Coordination). For any observations
What this assumption says is that the agent doesn't expect knowledge of a different policy-point
Assumption 5 (Knowledge of Decision Procedure).
Theorem 1. Assuming Fine-Grained Fairness, Limited Self-Modification, Faith in Argmax, Action Coordination, and Knowledge of Decision Procedure, UDT 1.0 does not strictly prefer any self-modifying action.
Proof. Suppose for contradiction that some self-modification
By Fine-Grained Fairness, the expectation of the self-modifying action is equal to the expectation of the corresponding non-self-modifying action
By definition of max, this expectation is at most the expectation of the best action which could be substituted for
By Faith in Argmax, this is at most the expected utility conditioning on the abstract statement that the best action will be chosen, rather than conditioning on the concrete best action:
By Action Coordination, I can drop one of the conditions inside the argmax:
By Knowledge of Decision Procedure, I can drop the argmax condition entirely:
Putting it all together, the self-modifying action is just as good as its non-self-modifying version:
This contradicts the initial assumption.
The generalized version
We can also have a generalized version of this proof. For this, we need to change some assumptions. Instead of Limited Self-Modification we have:
Assumption 6 (Hierarchical Self-Modification). For any self-modifying action
i.e. the order of the self-modifying action is well-defined, and
Assumption 7 (Finite Self-Modification). For any self-modifying action
Those assumptions ensure that all self-modifying actions eventually "roll out" (because the order is finite and the number of policy-points which are modified is finite) into non-self-modifying ones, without loops.
Instead of Fine-Grained Fairness, we have:
Assumption 8 (Faith in Fairness). For any self-modifying action
This is just as Fine-Grained Fairness, but we added that an action can modify multiple policy-points, and that conditioning on setting other policy-points does not interfere with the expectation equality.
We also need to generalize action coordination:
Assumption 9 (Generalized Action Coordination). For any
What this assumption says is that the agent doesn't expect knowledge of different policy-points
Theorem 2. Assuming Faith in Fairness, Hierarchical Self-Modification, Finite Self-Modification, Faith in Argmax, Generalized Action Coordination, and Knowledge of Decision Procedure, UDT 1.0 does not strictly prefer any self-modifying action.
Proof. Suppose for contradiction that some self-modification
By Hierarchical Self-Modification, this action has a finite order
By Faith in Fairness, the expectation of the self-modifying action is equal to the expectation of the corresponding non-self-modifying action
The actions
Applying Faith in Fairness to each of them (which is possible, since all other statements in the brackets are of the form of a conjunction of conditions on policy-points), the maximum order of actions in the policy-points being conditioned on decreases to
Then I get an expectation of the following form:
where
By definition of max, this expectation is at most the expectation of the best action which could be substituted for
By Faith in Argmax, this is at most the expected utility conditioning on the abstract statement that the best action will be chosen, rather than conditioning on the concrete best action:
By Generalized Action Coordination, I can drop the policy-point conditions inside the argmax:
By Knowledge of Decision Procedure, I can drop the argmax condition entirely:
The index of the
Putting it all together, the self-modifying action is just as good as its non-self-modifying version:
This contradicts the initial assumption.