So, if you haven't read the first two posts, do so now. 

In this post, we'll be going over the basic theory of belief functions, which are functions that map policies to sets of sa-measures, much like how an environment can be viewed as a function that maps policies to probability distributions over histories. Also, we'll be showing some nifty decision theory results at the end. The proofs for this post are in the following three posts (1,2,3), though it's inessential to read them and quite difficult.

Now, it's time to address desideratum 1 (dynamic consistency), and desideratum 3 (how do we formalize the Nirvana trick to capture policy selection problems) from the first post. We'll be taking the path where Nirvana counts as infinite reward, instead of counting as 1 reward forever. The latter option is cleaner mathematically, ties into learning theory better, and makes it clearer what a pseudocausal hypothesis is. But this post was long enough as-is, so we'll just be presenting the path where Nirvana counts as infinite reward, as that allows for more flexibility in encoding policy selection problems and a full tie-in with UDT.

Definition 1: Nirvana

A special observation $N$ which may occur at any time, counts as infinite reward, and brings the history to an end. An a-measure or sa-measure may be described as Nirvana-free when the set of histories ending in Nirvana has measure 0. The set of Nirvana-free a-measures is denoted as $NF$.

Our ambient mathematical setting is one where: At each point, there are finitely many discrete actions or observations available (no continuous observation spaces here, though the available space of actions or observations may vary depending on the history so far). In any situation, Nirvana may occur, and it ends the history. All policies are deterministic (refer back to the first post to see why this isn't much of an imposition).

We'll be working with bounded inframeasures (like a bounded infradistribution in all ways except normalization), instead of inframeasures, because it gets harder if we don't. Also, we'll be working within the cone of a-measures, but use the notion of upper-completion that uses the cone of sa-measures. This odd restriction is in place because Nirvana doesn't interact well with negative measures, as that would be $-\infty$ reward and Murphy minimizes your reward, which forces a lot of awkward kludges in various settings.

If we had Nirvana count as 1 reward forever, we could drop almost all of these issues, but we're trying to replicate UDT here. When cramming a UDT problem into this setting, we do it by hard-coding every possible policy into an environment that depends on a policy, which produces a family of environments, and add the constraint that you attain Nirvana if you violate the hard-coded policy.

In order for Murphy to go "dang, I'd better make the hard-coded policy match the agent's own" (simulating a policy-dependent environment), you need the loss incurred by the environment mispredicting you to be smaller than the gain in value from attaining Nirvana when the environment mispredicts you. Infinite reward gets this easily, while 1 reward forever isn't quite potent enough to stamp out mispredictions in all circumstances, it's limited to problem setups where predictors are predicting you in circumstances that don't have a tiny probability of arising and don't happen too far in the future.

Notation

Feel free to skip this part and refer back to it as necessary, or open up a seperate tab so you don't have to scroll all they way back up to this. A lot of it won't make sense now, it's just a reference. We have all the notation from the previous notation section, in addition to:

$h$: A finite history, ending in an observation. Similarly, $a$ is an action.

$\gamma$: A number in $[0,1)$, the time-discount parameter. This is used to define the distance between histories (for defining the KR-metric), as well as defining the distance between partial policies. We can also use $\gamma$ as a superscript in a utility function to specify the time discount.

${\Pi }^{+},\Pi$: The space of partial policies and full policies, respectively. All policies are deterministic. A partial policy could be thought of as a full policy, except that what it does may become undefined in some situations. Partial policies make a poset where the ordering is given by one partial policy being an extension of another. Full policies are maximal in this poset. There's also a distance metric between policies and partial policies, given by ${\gamma }^t$ where $t$ is the shortest time where they behave differently (whether by disagreeing with each other on what to do, or by one policy being defined where the other isn't)

${\pi }_{st},{\pi }_{pa},\pi$: A policy-stub, partial-policy, and full policy, respectively. Full policies are pretty obvious. A policy-stub is a partial policy which is guaranteed to be undefined after some finite time, though the time at which it becomes undefined may be history-dependent. A partial policy is the most inclusive concept, including both policy stubs and full policies, and is a tree of actions and observations that dead-ends at histories where you don't specify what is to be done after that history.

${\pi }_{\varnothing }$: The empty policy/empty stub. The maximally undefined partial policy that specifies absolutely nothing about what the policy does.

$F\left({\pi }_{pa}\right),F^{NF}\left({\pi }_{pa}\right)$: The outcome set of a partial policy. All the ending observation leaves on the tree given by ${\pi }_{pa}$. This has the property that no matter what, if you play that partial policy, you'll get one of those outcomes. If ${\pi }_{pa}$ is infinite down some paths, then full outcomes (infinite histories) down those paths are also taken to be in this set. $F^{NF}\left({\pi }_{pa}\right)$ is the same, but omits every Nirvana observation.

$M^a\left(F\right({\pi }_{pa}\left)\right),M^{sa}\left(F\right({\pi }_{pa}\left)\right)$: The a-measures and sa-measures over the set $F\left({\pi }_{pa}\right)$. An important part of this is that $M^{sa}\left(F\right({\pi }_{pa}\left)\right)$ can't assign negative measure to any Nirvana event, which is an important limitation, otherwise we could have $-\infty$ expected value and the agent would give up and cry since it plans for the worst-case.

$p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}$: The function with type signature $M^a\left(F\right({\pi }_{pa}^{hi}\left)\right)\to M^a\left(F\right({\pi }_{pa}^{lo}\left)\right)$ (or the Nirvana-free variant) defined as the pushforward of the function mapping each bitstring in $F\left({\pi }_{pa}^{hi}\right)$ to its unique prefix in $F\left({\pi }_{pa}^{lo}\right)$, which happens exactly when ${\pi }_{pa}^{hi}\ge {\pi }_{pa}^{lo}$. To be more concrete, the function that takes in a measure on 10-bit bitstrings and crunches it down to a measure on 8-bit prefixes is an instance of this sort of projection.

$e$: An environment.

$\Theta$: A belief function, mapping each ${\pi }_{pa}$ to a set of a-measures over $F\left({\pi }_{pa}\right)$. If these fulfill enough conditions, we call them hypotheses.

$NF$: The set of Nirvana-free a-measures for a given $F\left({\pi }_{pa}\right)$. Ie, the measure component doesn't assign nonzero measure to any Nirvana observation. Technically, there's a dependency on ${\pi }_{pa}$, but we suppress this in the notation since the definition of the set is pretty much the same for all the ${\pi }_{pa}$.(ie, everything without Nirvana in it)

${\lambda }^{\odot },b^{\odot }$: Minimal points of a-measures can be broken down as $(\lambda \mu ,b)$ where $\lambda ,b\ge 0$ and $\mu$ is a probability distribution. These are the minimal upper bounds on $\lambda$ and $b$ for all the $\Theta \left({\pi }_{pa}\right)$ sets, relative to a given $\Theta$.

${\pi }_{\neg h}$: Some partial policy which specifies everything the agent does except for what it does after history $h$.

$E_{\Theta \left({\pi }_{pa}\right)}\left(f\right)$: Like our standard expectations relative to a set of sa-measures, just remember that Nirvana means infinite reward, so this can be rephrased as $E_{\Theta \left({\pi }_{pa}\right)\cap NF}\left(f\right)$.

$P_{\Theta ,{\pi }_{\neg h}}^g\left(h\right)$: The quantity ${max}_{\pi >{\pi }_{\neg h}}{E}_{\Theta \left(\pi \right)}\left(1{★}^{h}g\right)-E_{\Theta \left({\pi }_{\neg h}\right)}\left(0{★}^{h}g\right)$. This is the rescaling term in updates, it can be (very) roughly thought of as the probability of getting the finite history $h$ relative to an off-h policy and a belief function.

$\overline{c.h}$: The closed convex hull of a set.

$M^a\left(\infty \right)$: The space of a-measures on all infinite histories (no Nirvana in here, it'd end the history early). Obeying a deterministic policy only let you reach a certain portion of this space.

$\{\le \odot \}$: Given a $\Theta$ and the associated ${\lambda }^{\odot }$ and $b^{\odot }$ upper bounds on the minimal points of all $\Theta \left({\pi }_{pa}\right)$, this is the set of all a-measures over $F\left({\pi }_{pa}\right)$ where $\lambda +b\le {\lambda }^{\odot }+b^{\odot }$. Again, like $NF$, there's an implicit dependence on ${\pi }_{pa}$ we're suppressing here. Used to clip away part of our set for some arguments.

$of$: An outcome function. A function mapping each ${\pi }_{pa}$ to a point in $\Theta \left({\pi }_{pa}\right)$ such that $p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}\left(of\right({\pi }_{pa}^{hi}\left)\right)=of\left({\pi }_{pa}^{lo}\right)$.

$SM$: An a-surmeasure or sa-surmeasure. Like an a-measure or sa-measure, except that the measure component can specify that some Nirvana events occur with arbitrarily-small-but-nonzero measure, where this quantity is denoted as $0^{+}$

$S$: A set of a-environments/a-survironments. They can be written as $(\lambda e,b)$, where $b$ is our constant term, $\lambda$ is "how much we care about the environment", and $e$ is an environment.

$I_{\ast }^{{\pi }_{pa}^{lo},{\pi }_{pa}^{hi}}$: There's also an $I_{\ast s}$ variant. This is the Nirvana injection up, and a measure is injected up by going "if we need to extend a piece of measure on a history $h$ to longer histories, we just say that $h$ is guaranteed to end in Nirvana after you take an action". $I_{\ast s}$ does the same thing, except that it also assigns every Nirvana event outside of $F\left({\pi }_{pa}^{lo}\right)$ that'd normally have 0 measure, $0^{+}$ measure instead.

$\uparrow ,\downarrow$: The functions mapping a ${\Theta }^{st}$ or ${\Theta }^{\omega }$ defined only over policy stubs or full policies to a $\Theta$ over all partial policies, via:

$\uparrow \left({\Theta }^{st}\right)\left({\pi }_{pa}\right):={\bigcap }_{{\pi }_{st}\le {\pi }_{pa}}\left(p{r}_{\ast }^{{\pi }_{pa},{\pi }_{st}}{)}^{-1}\right({\Theta }^{st}\left({\pi }_{st}\right))$

$\downarrow \left({\Theta }^{\omega }\right)\left({\pi }_{pa}\right):=\overline{c.h}\left({\bigcup }_{\pi \ge {\pi }_{pa}}p{r}_{\ast }^{\pi ,{\pi }_{pa}}\right({\Theta }^{\omega }\left(\pi \right)\left)\right)$

${\to }^{\omega },{\to }^{st}$: The functions taking a $\Theta$ and restricting it to be only defined over full policies or only defined over stubs.

${\to }^c,{\to }^{sc}$: The functions taking a hypothesis over stubs ${\Theta }^{st}$ that's pseudocausal/acausal and translating it to a causal/surcausal hypothesis over stubs, via:

${\to }^c\left(\Theta \right)\left({\pi }_{st}\right):=\overline{c.h}\left({\bigcup }_{{\pi }_{st}^{lo}\le {\pi }_{st}}{I}_{\ast }^{{\pi }_{st}^{lo},{\pi }_{st}}\right({\Theta }^{st}\left({\pi }_{st}^{lo}\right)\left)\right)$

And similar for ${\to }^{sc}$, just swap $I_{\ast }$ for $I_{\ast s}$.

${\to }^{NF}$: The function taking a causal/surcausal hypothesis $\Theta$ and translating it to a pseudocausal/acausal hypothesis by just intersecting everything with $NF$, ie, clipping away every a-measure with nonzero measure on some Nirvana event.

${\pi }_{\neg h}\bullet {\pi }_{pa}$: The partial policy made by taking ${\pi }_{\neg h}$ which specifies everything except what happens after observation $h$, and going "obey ${\pi }_{pa}$ after observation $h$". Just glue the two partial policies together.

$c$: The bookkeeping function which clips off the $h$ prefix from all the histories in your measure after you update on $h$. Technically, there's an $h$ dependence on this, but we suppress it in the notation.

$g^h$: The adjusted function $g$ over histories to compensate for clipping off the $h$ prefix on histories after updating, via $g^h\left(h^{\prime }\right):=g\left(h{h}^{\prime }\right)$.

$R$: When used as a superscript, it means something has been renormalized, when used as normal, it means regret.

$U$: Some utility function. $U^{\gamma }$ denotes a utility function with time discount parameter $\gamma$.

Foundations:

Definition 2: A-history

An a-history is a finite history ending with an action. A history is just a sequence of alternating actions and observations.

Definition 3: O-history

An o-history is a finite history ending with an observation, or an infinite history. The empty history will also count as an o-history.

Definition 4: Partial Policy

A partial policy ${\pi }_{pa}$ is a partial function from finite o-histories that can be continued with an action to actions, s.t. if ${\pi }_{pa}\left(h\right)$ is well-defined, and $h^{\prime }a$ is an a-history that is a prefix of $h$, then ${\pi }_{pa}\left(h^{\prime }\right)=a$. The set of these is denoted ${\Pi }^{+}$.

Note that in order to specify the behavior of a partial policy in a situation, you need to specify its behavior in all prefixes of the situation, which means you act in such a way that it's possible for the situation to come about. Also because you can only specify one action in any given situation, your action can't be specified on any o-history where the past actions don't match up with what your partial policy says you do.

Definition 5: Policy Stub

A policy stub ${\pi }_{st}$ is a partial policy that has a finite $n$ where if $|h|>n$, then ${\pi }_{st}\left(h\right)=\perp$. It is undefined beyond a certain time.

Definition 6: Full Policy

A full policy $\pi$ is a partial policy that cannot have its domain where it is defined enlarged, without ceasing to be a partial policy. The set of these is $\Pi$. This is the same as a policy in the usual sense of the word.

So, visualizing a partial policy as a big tree, policy stubs are the finite trees, full policies are the trees infinite down every branch, and partial policies are the all-encompassing category which also include trees that are infinite down some branches but not others.

Definition 7: Empty Policy

The empty policy ${\pi }_{\varnothing }$ is the partial policy/policy stub that is completely undefined. For all o-histories $h$, even including the empty history, ${\pi }_{\varnothing }\left(h\right)=\perp$

We can equip ${\Pi }^{+}$ with a distance metric. The distance metric is ${\gamma }^{t({\pi }_{pa},{\pi }_{pa}^{\prime })}$, where $\gamma <1$, and $t({\pi }_{pa},{\pi }_{pa}^{\prime })$ is the "time of first difference", the length of the shortest o-history where the action of ${\pi }_{pa}$ doesn't match up with the action of ${\pi }_{pa}^{\prime }$, or where one of the ${\pi }_{pa}$ is defined and the other one isn't.

We can also equip ${\Pi }^{+}$ with a partial order. The partial order is ${\pi }_{pa}^{hi}\ge {\pi }_{pa}^{lo}$ if the two policies never disagree on which action to take, and ${\pi }_{pa}^{hi}$ is defined on more histories than ${\pi }_{pa}^{lo}$ is. $inf$ can be taken of arbitrary collections of partial policies (intersection), and $sup$ (union) can be taken of arbitrary collections of partial policies as long as they're all below some full policy $\pi$.

The full policies are the maximal elements in this poset, the empty policy is the bottom element in this poset, and every partial policy ${\pi }_{pa}$ is associated with a sequence of stubs ${\pi }_{pa}^n$ that are like "specify all of what this partial policy does up until time $n$, then be undefined afterwards". The chain ${\pi }_{pa}^n$ ascends up towards ${\pi }_{pa}$, and has the property that any stub ${\pi }_{st}\le {\pi }_{pa}$ has some ${\pi }_{pa}^n$ where ${\pi }_{st}\le {\pi }_{pa}^n$.

Outcome Sets and Projection Mappings

The "outcome set" of a partial policy, $F\left({\pi }_{pa}\right)$, is... well, let's start with a visualization and then give the formal definition.

Consider the branching tree of alternating actions and observations. The partial policy fixes a subtree of this tree, where every child of an observation node is an action given by ${\pi }_{pa}$, and you extend no further if the policy becomes undefined at an observation. Then the outcome set $F\left({\pi }_{pa}\right)$ would be all the leaves and infinite paths of that tree. If you follow the behavior of the partial policy, your history is guaranteed to go through one of the leaves. Thus, if your partial policy interacts with an environment, you'd get a probability distribution over $F\left({\pi }_{pa}\right)$ (for the history is guaranteed to have something in $F\left({\pi }_{pa}\right)$ as a prefix). Observe the picture.

Definition 8: Outcome Set

The outcome set $F\left({\pi }_{pa}\right)$ is the set of o-histories $h$ that are not in the domain of ${\pi }_{pa}$, but all prefix a-histories $h^{\prime }a$ of $h$, have $a={\pi }_{pa}\left(h^{\prime }\right)$. By this definition, $F\left({\pi }_{\varnothing }\right)$ contains one element, the empty history. $F^{NF}\left({\pi }_{pa}\right)$ is the subset of $F\left({\pi }_{pa}\right)$ consisting of all o-histories which don't end in Nirvana.

$F\left({\pi }_{pa}\right)$ and $F^{NF}\left({\pi }_{pa}\right)$ will take the role of our arbitrary compact metric spaces $X$ that we defined infradistributions over. The metric is $d(h,h^{\prime })={\gamma }^{t(h,h^{\prime })}$, where $\gamma <1$ and $t(h,h^{\prime })$ is the first timestep where the histories differ. So two histories are close if they agree for a long time before differing.

However, there's a technicality we need. While $M^a\left(F\right({\pi }_{pa}\left)\right)$ and $M^a\left(F^{NF}\right({\pi }_{pa}\left)\right)$ and $M^{sa}\left(F^{NF}\right({\pi }_{pa}\left)\right)$ work exactly as you'd expect, $M^{sa}\left(F\right({\pi }_{pa}\left)\right)$ can't contain signed measures that assign negative mass to a Nirvana event. This is because a positive functional (that respects Nirvana) would think that such a measure may have $-\infty$ value. And requiring that the amount of Nirvana-measure is equal or greater than the amount of negative-Nirvana measure interacts badly with updates. This issue causes minor kludges all over the place, we'll reiterate that Nirvana counting as 1 reward is much more elegant.

We'll also need the notion of a projection (this is going to be our analogue of the $g_{\ast }$ mapping from the basic theory).

Definition 9: Projection Mapping

If ${\pi }_{pa}^{hi}\ge {\pi }_{pa}^{lo}$, then the function $p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}:M^{sa}\left(F\right({\pi }_{pa}^{hi}\left)\right)\to M^{sa}\left(F\right({\pi }_{pa}^{lo}\left)\right)$ is defined by:

$p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}(m,b):=(m^{\prime },b)$

Where, if $h$ is a prefix of some outcome in $F\left({\pi }_{pa}^{lo}\right)$, then $m^{\prime }\left(h\right)=m\left(h\right)$, where $m\left(h\right)$ and $m^{\prime }\left(h\right)$ should be understood to mean "measure assigned to the outcome having prefix $h$"  This function also maps a-measures to a-measures and Nirvana-free stuff to Nirvana-free stuff.

To expand more on this, if ${\pi }_{pa}^{hi}\ge {\pi }_{pa}^{lo}$, then there's a natural surjection from $F\left({\pi }_{pa}^{hi}\right)$ to $F\left({\pi }_{pa}^{lo}\right)$ (which also acts as a surjection from $F^{NF}\left({\pi }_{pa}^{hi}\right)$ to $F^{NF}\left({\pi }_{pa}^{lo}\right)$). Every history in $F\left({\pi }_{pa}^{hi}\right)$ has a unique prefix in $F\left({\pi }_{pa}^{lo}\right)$, so just send a history to its unique prefix. This is $p{r}^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}$.

To visualize it, take your tree induced by ${\pi }_{pa}^{lo}$. $F\left({\pi }_{pa}^{lo}\right)$ is the coat of observation leaf nodes and all the infinite paths. Extending ${\pi }_{pa}^{lo}$ requires the new tree ${\pi }_{pa}^{hi}$ to poke out through some observation leaf nodes in $F\left({\pi }_{pa}^{lo}\right)$, so every leaf node in $F\left({\pi }_{pa}^{hi}\right)$ is attached to a part of the ${\pi }_{pa}^{hi}$ tree that pokes out through the original coat of leaf nodes $F\left({\pi }_{pa}^{lo}\right)$ somewhere.

$p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}$ is just the pushforward mapping induced by this, like how we got a function $g_{\ast }$ from sa-measures to sa-measures from a function $g:X\to Y$ back in the last post. $p{r}_{\ast }$ is the exact function you'd come up with if you went "given a measure on 10-bit bitstrings, how do I craft a measure on 7-bit prefixes" and generalized it.

Some important properties of this are that it's identity when mapping from a partial policy to itself, linear, preserves the $\lambda$ and $b$ value of an a-measure when you project it down, and it commutes. It doesn't matter what sequence of projections you take to get from a high partial policy to a low partial policy, they'll all be equal.

Definition 10: A-Environment

An a-environment is a point written as $(\lambda e,b)$, where $b,\lambda \ge 0$, and $e$ is an environment. This is parallel to an a-measure.

The intuition for this is that $\lambda$ tells you how much you care about the environment $e$/the probability of the environment existing in the first place if $\lambda <1$, and $b$ is the utility you're guaranteed/portion of expected utility you get if the environment doesn't exist.

We can think of querying an a-environment with a partial policy to get an a-measure via $\left(\lambda \right({\pi }_{pa}\cdot e),b)$, where ${\pi }_{pa}\cdot e$ is the probability distribution over $F\left({\pi }_{pa}\right)$ induced by ${\pi }_{pa}$ interacting with the environment. We'll use this as an intuition pump a lot. In fact, environments are actually equivalent to functions mapping partial policies to probability distributions over actions. One direction is ${\pi }_{pa}\mapsto {\pi }_{pa}\cdot e$, and in the reverse direction, if you've got a function $\Theta$ mapping partial policies to a probability distribution over outcomes,  then as long as $p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}\left(\Theta \right({\pi }_{pa}^{hi}\left)\right)=\Theta \left({\pi }_{pa}^{lo}\right)$ when ${\pi }_{pa}^{hi}\ge {\pi }_{pa}^{lo}$, then it corresponds to an environment. This is the motive for the next part.

Belief Functions

Definition 11: Belief Function

A belief function $\Theta$ is a function that assigns all ${\pi }_{pa}$ to a nonempty set of a-measures over $F\left({\pi }_{pa}\right)$ (or the Nirvana-free variant).

The reason we're using a-measures instead of sa-measures is because we really don't need negative measures except for defining the upper completion, and negative measures really don't interact well with Nirvana being infinite reward (a drawback of this approach)

From the a-environment view, you can consider taking a set of a-environments, letting them all interact with a partial policy, and then plotting the resulting a-measures over $F\left({\pi }_{pa}\right)$ you get, that'd be your $\Theta$ induced by a set of a-environments. But we'll just work with belief functions $\Theta$ more generally and later build up the tools to view them as a set of a-environments. We'll also abuse notation and use $NF$ (Nirvana-free) as an abbreviation for $M^a\left(F^{NF}\right({\pi }_{pa}\left)\right)$ sometimes, when we don't want to specify an ambient space, and are just doing an intersection to clip away Nirvana.

Also, $E_{\Theta \left({\pi }_{pa}\right)}\left(f\right)=E_{\Theta \left({\pi }_{pa}\right)\cap NF}\left(f\right)$, because any probability of Nirvana counts as infinite value, so Murphy will only pick from Nirvana-free a-measures.

Belief Function Conditions

Let's go over the various conditions for a belief function to behave well. The first 6 are analogues of the corresponding conditions for an infradistribution, and the last 3+2 are special conditions for how the different $\Theta \left({\pi }_{pa}\right)$ sets interact with each other.

Condition 1: Nirvana-free Nonemptiness: $\forall {\pi }_{pa}:\Theta \left({\pi }_{pa}\right)\cap NF\ne \varnothing$

This is parallel to condition 1: Nonemptiness, for infradistributions.

It's nontrivial in the Nirvana-containing case, because it says "regardless of your policy, Murphy can find an environment to play that avoids Nirvana", as should be the case if we don't want the agent ignoring all else to optimize its chances of getting infinite reward which never happens.

Condition 2: Closure: $\forall {\pi }_{pa}:\Theta \left({\pi }_{pa}\right)=\overline{\Theta \left({\pi }_{pa}\right)}$

This is parallel to condition 2: Closure, for infradistributions.

This also works with the Nirvana-free interpretation of $\Theta$, because $M^a\left(F^{NF}\right({\pi }_{pa}\left)\right)$ is closed.

Condition 3: Convexity: $\forall {\pi }_{pa}:\Theta \left({\pi }_{pa}\right)=c.h\left(\Theta \right({\pi }_{pa}\left)\right)$

This is parallel to condition 3: Convexity, for infradistributions.

Again, $M^a\left(F^{NF}\right({\pi }_{pa}\left)\right)$ is convex so we can do this in that setting too. $c.h$ is the convex hull.

Condition 4: Nirvana-Free Upper Completeness:

$\forall {\pi }_{pa}:\Theta \left({\pi }_{pa}\right)\cap NF=\left(\right(\Theta \left({\pi }_{pa}\right)\cap NF)+M^{sa}(F^{NF}\left({\pi }_{pa}\right)\left)\right)\cap M^a\left(F\right({\pi }_{pa}\left)\right)$

This is roughly parallel to condition 4: Upper Completeness, for infradistributions. This says that the Nirvana-free part of $\Theta \left({\pi }_{pa}\right)$ should be upper-complete w.r.t. the cone of Nirvana-free sa-measures. This is a bit of a hack, because Nirvana counts as infinite reward.

You probably have several questions. The first one: why do we restrict to a-measures? Well, our minimal points should be a-measures (by Condition 5: Positive Minimals for infradistributions), and those are the only thing that matters, so it's no big deal that we trim off all the sa-measures. Plus, some later tools like Nirvana injections don't work well with sa-measures.

Second question: Why do we need the cone of Nirvana-free sa-measures? Can't we just add the cone of Nirvana-free a-measures? Well, it's possible to add an sa-measure to an a-measure and get an a-measure, so just adding the cone of a-measures wouldn't capture enough points, we could add more.

Third question: Why do we require everything to be Nirvana-free? Can't we just require upper completion in general? Well, there's a weird thing where the projection of an upper-complete set isn't necessarily upper-complete when there's Nirvana about. But, in the Nirvana-free setting, the projection of an upper-complete set is upper-complete. This feature messes with several downstream things. Murphy will only pick from Nirvana-free a-measures, so it's no big loss.

Again, we see the pattern that having Nirvana be infinite forces a bunch of weird minor issues all over the place.

Condition 5: Bounded Minimals:

$\exists {\lambda }^{\odot },b^{\odot }:\forall {\pi }_{pa}:(\lambda \mu ,b)\in \left(\Theta \right({\pi }_{pa}){)}^{min}\to \lambda \le {\lambda }^{\odot }\wedge b\le b^{\odot }$

This is roughly parallel to condition 6: Bounded-Minimals for infradistributions. Note that said bound must be uniform among all the $\Theta \left({\pi }_{pa}\right)$ sets. We need to bound the $b$ term uniformly as well to make some proofs work out.

Condition 6: Normalization: ${min}_{\pi }{E}_{\Theta \left(\pi \right)}\left(0\right)=0\wedge {max}_{\pi }{E}_{\Theta \left(\pi \right)}\left(1\right)=1$

This is not the standard renormalization, but analogous. It also has the bad property that a mixture of normalized belief functions is not normalized. In our defense, it's all scale-and-shift anyways, you can just renormalize after mixing if you want.

Why is this defined in such a way? Well, it's because we're actually dealing with a bunch of different sets of a-measures, one for each partial policy, and since they're different, we can't get a guarantee that they're all normalized at once. You can't do individual normalizations, that'd break dynamic consistency. So we do a sort of collective normalization to $\Theta$ as a whole, to preserve the natural analogue of the normalization condition in this setting. Worst-case value should be 0 and best-case value should be 1, and this attains that.

Now, we can move on to three additional conditions on how the various $\Theta \left({\pi }_{pa}\right)$ sets link together, instead of merely checking the usual infradistribution conditions for individual $\Theta \left({\pi }_{pa}\right)$.

Consistency and Extreme Points

Condition 7: Consistency:

$\forall {\pi }_{pa}:\Theta \left({\pi }_{pa}\right)={\bigcap }_{{\pi }_{st}\le {\pi }_{pa}}\left(p{r}_{\ast }^{{\pi }_{pa},{\pi }_{st}}{)}^{-1}\right(\Theta \left({\pi }_{st}\right))$

$\forall {\pi }_{pa}:\Theta \left({\pi }_{pa}\right)=\overline{c.h}\left({\bigcup }_{\pi \ge {\pi }_{pa}}p{r}_{\ast }^{\pi ,{\pi }_{pa}}\right(\Theta \left(\pi \right)\left)\right)$

This seems a bit mystifying, but it pretty much says that the behavior of the finite levels (the policy-stubs ${\pi }_{st}$ which only specify things up to a finite time), and the infinite levels (the full policies $\pi$), uniquely pin each other down. This condition lets you recover $\Theta$ uniquely via only knowing it on policy-stubs or only knowing it on full policies.

The intuition for the first equality will be taken from the view where $\Theta$ tells you what you get via querying a set of a-environments with a partial policy and recording the a-measures you get. So, to begin with, if $\pi \ge {\pi }_{pa}$, then querying an a-environment with $\pi$ and projecting your result down via $p{r}_{\ast }^{\pi ,{\pi }_{pa}}$, gets you the same thing as querying that environment with ${\pi }_{pa}$. And if the first query gave you a Nirvana-free thing, the second one must have done so too because $\pi \ge {\pi }_{pa}$. Therefore, $p{r}_{\ast }^{\pi ,{\pi }_{pa}}\left(\Theta \right(\pi \left)\right)\subseteq \Theta \left({\pi }_{pa}\right)$. Also, if the set of a-environments is convex (the corresponding property to all the $\Theta \left({\pi }_{pa}\right)$ being convex), we should be able to mix a-environments, which mixes the a-measures you get via a query in the same way. So,

$\Theta \left({\pi }_{pa}\right)\supseteq c.h\left({\bigcup }_{\pi \ge {\pi }_{pa}}p{r}_{\ast }^{\pi ,{\pi }_{pa}}\right(\Theta \left(\pi \right)\left)\right)$

And also, $\Theta \left({\pi }_{pa}\right)$ is closed, so $\Theta \left({\pi }_{pa}\right)$ contains the closed convex hull as well.

In the Nirvana-containing case, we can go "hey, if there's a point in $\Theta \left({\pi }_{pa}\right)$ that the closed convex hull doesn't nab, can't we just query the associated superenvironment with some random $\pi \ge {\pi }_{pa}$ to get something in $\Theta \left(\pi \right)$ that projects down to our point of interest?" And this gets you equality.

If we slice out the Nirvana, this equality is a significantly stronger condition. We could imagine taking an a-environment that's Nirvana-free for a long time, but past a certain point, regardless of policy, Nirvana is guaranteed. The a-measures made by querying this a-environment with all partial policies would show up in the small policy-stubs (being Nirvana-free), but because Nirvana eventually shows up regardless of policy, the a-measures would get clipped out of our $\Theta \left({\pi }_{pa}\right)$ sets because they have Nirvana in them and we're only recording Nirvana-free a-measures.

So, in the Nirvana-free case, this condition corresponds to saying that, given some Nirvana-free a-measure produced by a partial policy interacting with an a-environment, there should be some way to extend the partial policy to a full policy that's Nirvana-free as well. Or, if that can't be done, the a-environment should be a mixture of a-environments where you can extend your policy in a Nirvana-free way.

In the Nirvana-free case, this "$\Theta \left({\pi }_{pa}\right)$ is the closed convex hull of projections from above" property is basically saying "don't clog $\Theta$ of the partial policy with extraneous crap that didn't arise from appropriate Nirvana-free a-measures for full policies".

After all, we want to recover the whole thing from either the behavior of $\Theta$ on full policies, or the policy-stubs, and adding extra Nirvana-free stuff into the policy-stubs would mess with our ability to do this.

Now, let's look at the other one, $\Theta \left({\pi }_{pa}\right)={\bigcap }_{{\pi }_{st}\le {\pi }_{pa}}\left(p{r}_{\ast }^{{\pi }_{pa},{\pi }_{st}}{)}^{-1}\right(\Theta \left({\pi }_{st}\right))$

What's the motive for this one? Well, let's cover both directions. In one direction, if you were like "this intersection of preimages of projections doesn't have enough points, I'm gonna throw in another point $M$ outside the intersection", then there'd be some stub ${\pi }_{st}\le {\pi }_{pa}$, where your added point would project down to be outside of $\Theta \left({\pi }_{st}\right)$, which, oops, can't be done. Remember, if you got your extra point $M$ via querying an a-environment with ${\pi }_{pa}$, then you should be able to query the same a-environment with ${\pi }_{st}$ which is shorter, to get the projection of $M$ which had better be in $\Theta \left({\pi }_{st}\right)$. But it's not.

In the other direction, this is sort of an "instantiating points at infinity" condition. A point in that intersection would project down to $\Theta \left({\pi }_{st}\right)$ for all appropriate stubs. We've got a sequence of points in those sets, one for each stub. So, you can't actually detect the addition of these points at infinity, because their finite-time behavior will always flawlessly match up with some preexisting thing. Remember, we want to add as many points as we can that don't affect any observable behavior.

Thus, the consistency condition can be viewed as a mix of:

1: Projecting $\Theta \left({\pi }_{pa}\right)$ down should make a subset of $\Theta \left({\pi }_{st}\right)$ because if you can query an a-environment with a long policy, you can also query it with a short one.

2: The finite levels aren't clogged with extra crap that didn't come from infinite levels.

3: A point is present at infinity if it's present at all the finite levels, because no finite-time observations can make any headway towards ruling that point out.

Condition 8: Extreme Point Condition:

$(M\in (\Theta \left({\pi }_{st}\right){)}^{xmin}\cap NF)\to (\exists \pi >{\pi }_{st},M^{\prime }:M^{\prime }\in \Theta (\pi )\cap NF\wedge p{r}_{\ast }^{\pi ,{\pi }_{st}}(M^{\prime })=M)$

$\left(\Theta \right({\pi }_{st}){)}^{xmin}$ is the set of extreme minimal points of $\Theta \left({\pi }_{st}\right)$.

To unpack this, an extreme point of a set $B$ is a point that can't be written as a mixture of distinct points in $B$. Given a tetrahedron, the vertices would be extreme points, and given a sphere, the surface would be the extreme points.

So this says that, for stubs, if a point $M$ is an extreme point, minimal, and Nirvana-free, then it's the projection of a Nirvana-free point from a full policy above the stub.

In the Nirvana-free case, we get this for free from consistency. An extreme point can't be written as a mixture of other points, so it wasn't introduced when we took the convex hull of the union of projections (closure is more fiddly and we'll be ignoring it), so it had to come from the projection of a point from above.

However, in the Nirvana-containing case, this condition has two interpretations. The first is that shearing off all the a-measures with Nirvana occurring in them should get you a well-formed hypothesis. Our first condition, Nirvana-Free Nonemptiness, can be thought of as a condition in that vein, which ensures that shearing off the Nirvana doesn't leave you with an empty set. This condition is similar. Shearing off all the Nirvana had better preserve consistency! And this condition ensures exactly that.

Another characterization is that Nirvana cannot become inevitable without already happening. Minimal points are the essential elements of an inframeasure that can't be stripped away without affecting the expectation values, and extreme points are those that cannot be introduced by taking convex hull if they weren't already present. So, this condition says that if a point is minimal and extreme (can't be made by mixing and can't be removed without affecting anything else), then if Nirvana/infinite reward is guaranteed to have a shot at occurring in the future, then we might as well assume it has already occurred and said point can't be Nirvana-free.

Hausdorff-Continuity

Let's introduce some new concepts for this one.

Definition 12: A-Measures on Nirvana-Free Histories

$M^a\left(\infty \right)$ is the set of a-measures on infinite o-histories. A finite history you can extend doesn't count. A finite history that ends with Nirvana doesn't count. An infinite history counts. This is all histories of this form, not just the ones compatible with some policy or other.

Definition 13: Clip Set

Given a belief function $\Theta$ that fulfills bounded-minimals, there are ${\lambda }^{\odot }$ and $b^{\odot }$ upper bounds on the $\lambda$ and $b$ values of minimal points. $\{\le \odot \}$ is the set of a-measures $(\lambda \mu ,b)$ where $\lambda +b\le {\lambda }^{\odot }+b^{\odot }$

Definition 14: Preimage to Infinity

$(p{r}_{\ast }^{\infty ,{\pi }_{pa}}{)}^{-1}$ is the function that maps a set $B$ in $M^a\left(F^{NF}\right({\pi }_{pa}\left)\right)$ to the set of a-measures in $M^a\left(\infty \right)$ with a measure of 0 on histories which don't have a history in $F^{NF}\left({\pi }_{pa}\right)$ as a prefix (so they can project down onto $M^a\left(F^{NF}\right({\pi }_{pa}\left)\right)$ safely), and which project down to $B$.

This allows us to take the preimage of all our $\Theta \left({\pi }_{pa}\right)\cap NF$ sets and have them all coexist in the same space, for ready comparison.

Now, let's define our next condition.

Condition 9: Hausdorff Continuity: ${\pi }_{pa}\mapsto \left(p{r}^{\infty ,{\pi }_{pa}}{)}^{-1}\right(\Theta \left({\pi }_{pa}\right)\cap NF\cap \{\le \odot \})$ is continuous.

"Wait a minute", you might ask, "this maps partial policies to sets, what notion of continuity are you using here??"

Well, given two sets $B$ and $B^{\prime }$ in a metric space, the Hausdorff-distance between these sets is the maximum distance you'd have to travel to get to a point in one set, if you started at an arbitrary point in the other set. Two sets being close in Hausdorff-distance means that every point from $B$ is close to a point in $B^{\prime }$, and vice-versa. From earlier, we've got a distance metric between partial policies. Further, because ${\Pi }^{+}$ (the set of partial policies) is compact, said function isn't just continuous, but uniformly continuous.

Why do we need Nirvana-freeness? Well, $M^a\left(\infty \right)$ doesn't include any Nirvana in it, so projecting it down can't make any a-measure that has Nirvana in it.

Why must we clip off part of our set? For full policies, $\left(p{r}_{\ast }^{\infty ,\pi }{)}^{-1}\right(\Theta \left(\pi \right)\cap NF)=\Theta (\pi )\cap NF$

Now, $\Theta \left(\pi \right)\cap NF$ is upper-closed w.r.t $F^{NF}\left(\pi \right)$. So there's some points in that set with a gigantic amount of measure on infinite histories compatible with $\pi$, and 0 everywhere else. Moving to a slightly different ${\pi }^{\prime }$, its associated set has points with a gigantic amount of measure on infinite histories compatible with ${\pi }^{\prime }$, and 0 everywhere else. Basically, the thing going wrong if we don't clip off the upper completion "tail" of our set is that the upper-completion tail is pointing in slightly different directions for different policies, so the Hausdorff-distance gets arbitarily large in the upper-completion tails. So we need to clip off that part. This doesn't clip off any minimal points, which are the important ones and everything else is extraneous.

This condition may seem quite mysterious. Speaking roughly, it says that for a Nirvana-free a-measure $M\in \Theta \left(\pi \right)\cap NF$, if we make a perturbation of our policy at a very late time to get ${\pi }^{\prime }$, there should be some nearby Nirvana-free a-measure $M^{\prime }\in \Theta \left({\pi }^{\prime }\right)\cap NF$.

This is a continuity condition on Murphy. If your policy trembles a bit to some nearby policy, then Murphy should only have to change the environment a little bit to get something Nirvana-free for the new policy. Another thing that may make the situation clearer is providing an example of what this condition rules out.

Let's say you're doing iterated Newcomb. If Omega fills the box only if it predicts you 1-box on the next 100 rounds, that's permitted by this condition. Altering your policy on round 1 million only alters the environment past round 1 million-100, which isn't much of a change w.r.t the KR metric we're using (it's insensitive to changes at late times). Omega can even look at your full policy, as long as really distant changes in your policy only have a small influence on the probabilities of what happens now, that'd still fulfill this condition. What this does forbid is the variant of iterated Newcomb where Omega fills the box only if it predicts you always 1-box. Because an arbitrarily late policy perturbation leads to a big change in what happens now.

So, despite looking hard to fulfill, this condition is an "acausal influences from very distant futures are weak" condition which is fairly natural. Another way of thinking about it is that you should be able to figure out what a policy-dependent environment is doing by just making bounded precommitments, instead of precommitments that extend for all time.

Why do we need this condition? Well, to prove that there's an optimal policy. Surprisingly enough, it's also essential for the proof of the consistency condition when we're trying to recover the whole $\Theta$ from a belief function defined only on stubs or full policies.

So, the Hausdorff-continuity condition is more natural than it seems, and essentially says that extremely late policy changes don't change your situation now very much.

Causal, Pseudocausal, Acausal

But there's two more conditions, to define special sorts of hypotheses! Causal, Pseudocausal, and Acausal respectively. Causal hypotheses can have Nirvana in them, Pseudocausal and Acausal must be Nirvana-free. Acausal doesn't have any extra conditions on it besides the previous nine and being Nirvana-free, but Causal and Pseudocausal do need one extra condition each.

Definition 15: Outcome Function

An outcome function $of$ is a function mapping each ${\pi }_{pa}$ to a point in $M^a\left(F\right({\pi }_{pa}\left)\right)$, such that:

$\forall {\pi }_{pa}^{lo},{\pi }_{pa}^{hi}\ge {\pi }_{pa}^{lo}:p{r}_{\ast }^{{\pi }_{pa}^{hi},{\pi }_{pa}^{lo}}\left(of\right({\pi }_{pa}^{hi}\left)\right)=of\left({\pi }_{pa}^{lo}\right)$

Condition C: Causality

$\forall {\pi }_{pa},M\in \Theta \left({\pi }_{pa}\right)\exists of:of\left({\pi }_{pa}\right)=M\wedge (\forall {\pi }_{pa}^{\prime }:of({\pi }_{pa}^{\prime })\in \Theta ({\pi }_{pa}^{\prime }\left)\right)$

The intuition for this is that, if you fix an a-environment, and queried it with all possible partial policies, you'd produce an outcome function. Similarly, an outcome function uniquely picks out an a-environment because you know what occurs in all possible situations and all the projections commute. So, the causality condition indirectly says "every point in all of of your belief function sets arose from querying an a-environment". This lets us easily go from the belief function to a set of a-environments, and similarly, querying a set of a-environments with partial policies will make a causal hypothesis.

Condition P: Pseudocausality

$\forall {\pi }_{pa},{\pi }_{pa}^{\prime }:\left(\right(m,b)\in \Theta ({\pi }_{pa})\wedge \text{supp}(m)\subseteq F^{NF}({\pi }_{pa}^{\prime }\left)\right)\to (m,b)\in \Theta \left({\pi }_{pa}^{\prime }\right)$

This is the Nirvana-free analogue of causality. Let's say we've got some a-environment that we query with ${\pi }_{pa}$, and the resulting distribution is Nirvana-free, and supported over $F\left({\pi }_{pa}^{\prime }\right)$. Now, if we query said a-environment with ${\pi }_{pa}^{\prime }$ instead... we don't have to worry about introducing Nirvana! Yup, ${\pi }_{pa}^{\prime }$ behaves a bit differently. But, it only behaves differently than ${\pi }_{pa}$ on histories that have 0 probability of happening, so it should induce the exact same a-measure. Causality implies this condition, but this is weaker than causality. This is the property you'd get if you were taking a set of a-environments and querying it and reporting only the Nirvana-free a-measures. The special property of this is that, if pseudocausality is fulfilled, we can add Nirvana in such a way to craft a causal hypothesis (which is linked to actual sets of a-environments)

Recap! Ok, so our conditions are:

1: Nirvana-free Nonemptiness

This is easy, we shouldn't have the agent chasing after infinite reward.

2,3,4: Closure, Convexity, Nirvana-Free Upper Completion

These collapse equivalence classes of a-measures that have the exact same behavior, and are parallel to the conditions for inframeasures.

5: Minimal-Boundedness

Again, this is parallel to the condition for bounded inframeasures, just adding in a $b$ bound as well. It makes proofs much nicer.

6: Normalization

We impose a sort of collective normalization for the belief function as a whole, parallel to the corresponding condition for an infradistribution. This is not preserved when we mix belief functions.

7: Consistency

You should be able to recover the finite behavior of $\Theta$ from the infinite behavior of $\Theta$, or vice-versa. Projecting big partial policies down to low levels makes subsets, the finite levels aren't clogged with extraneous crap, the infinite levels have all the points that match up with finite behavior.

8: Extreme Point Condition

Shearing the Nirvana off a consistent Nirvana-containing hypothesis should produce a consistent hypothesis.

9: Hausdorff-Continuity

Policy changes at extremely late times don't have much effect on what happens now.

C: Causality

Your belief function is induced by querying a set of a-environments.

P: Pseudocausality

Your belief function is induced by querying a set of a-environments and only recording the Nirvana-free results.

Definition 16: Causal/Pseudocausal/Acausal/Surcausal Hypothesis

A hypothesis is a belief function $\Theta$ fulfilling the 9 conditions. A causal hypothesis may include Nirvana and fulfills Causality. A pseudocausal hypothesis is Nirvana-free and fulfills Pseudocausality. An acausal hypothesis is Nirvana-free. A surcausal hypothesis is a causal hypothesis over a-surmeasures (to be defined)

What's the essential difference between pseudocausal and acausal hypotheses? Well, it's a natural dividing line between two importantly distinct classes of decision-theory problems. In the former, no matter how you're mispredicted, you must have some chance of actually entering the situation where you're mispredicted. In the other class, you could be mispredicted and then locked out so you can't show the misprediction is wrong. More on this in the next section.

As we'll show later, you can turn any pseudocausal hypothesis into an equivalent causal one by adding Nirvana appropriately, and turn any causal hypothesis into an actual set of a-environments. But we have a hard time viewing acausal hypotheses as coming from a set of a-environments, at least without further work. So, the next section will be about poking at a toy case where we have an acausal hypothesis we can't turn into a causal one, and seeing how we have to adapt things to view an agent as believing that it's in some set of environments, instead of the agent working purely with belief functions that don't have a representation as a set of environments.

Acausal Hypotheses and Surmeasures

Let's take the example of Transparent Newcomb, where Omega fills the transparent box with a hundred dollars if it predicts you'd 1-box upon seeing the transparent box filled, and has $ϵ$ probability of making a mistake (the other box contains 1 dollar). This is a policy-dependent environment, which depends on your policy, as well as your action and the past history. The way we can encode this as a set of environments is by hardcoding all possible policies into the policy slot to make a set of environments, and send you to Nirvana if you ever act contrary to the hardcoded policy. This forces Murphy to make the hardcoded policy match your own, lest you go to Nirvana. The $ϵ$ chance of error guarantees that you have a shot at Nirvana if the hardcoded policy doesn't match your own. Translating this set of environments to a Nirvana-free belief function yields a pseudocausal hypothesis. XOR blackmail and Counterfactual Mugging are also pseudocausal hypotheses.

But what about Transparent Newcomb with a perfect predictor? In that case, we have a very serious problem with our standard way of viewing a policy-dependent environment as a set of environments via hardcoding in all policies and sending you to Nirvana if you violate the hardcoded policy. Maybe you 1-box upon seeing a full box, 2-box upon seeing it empty, but the hardcoded policy in the environment is "you 2-box in both cases". You're stuck 2-boxing and getting 1 dollar, you can never get into a situation where you get the Nirvana reward from calling out Omega on its misprediction. In fact, both "1-box upon full, 2-box upon empty" and "always 2-box" are optimal policies here, so this encoding of a policy-dependent environment doesn't let us get UDT behavior. If we don't want to permanently give up on viewing belief functions as sets of a-environments, how can we go forward?

Well, there's two ways to deal with this. One is encoding $ϵ$-exploration appropriately, the other doesn't require $ϵ$-exploration and is more interesting.

For the first one, assume a situation where you have an "intended policy" (deterministic), and you have an internal sense of what you intended to do, but maybe exploration overwrites that. Omega has a hard-coded prediction of your policy, and an exploration coin is flipped on each round. If the coin says "act as normal", things proceed as normal, if the coin says "explore", then Omega does the opposite of what it would normally do re: box-filling or not since it knows you'll do the opposite of what you'd normally do. Then, if the action you intended upon seeing the box's condition doesn't match up with Omega's prediction of what you intended, you go to Nirvana. In either case, the real action is determined by the exploration coin and what you intended to do. If we try going "Omega's prediction is that you intend to 2-box regardless of whether the box is full or empty", and our true intent is 1-boxing if full, 2-boxing if empty, and the coin says to explore, then Omega goes "they do the opposite of what they'd normally do, better fill the box". Then, as you reach for both boxes (because of exploration), the mismatch between your intent of 1-boxing and Omega's prediction that you intended to 2-box manifests and you attain Nirvana in the exploration step. This particular way of setting up things guarantees some probability of getting into a situation where you prove Omega wrong if there's a mismatch between your policy and the hardcoded policy, so we can turn it into a set of environments. But now, let's look at a different way of resolving the problem.

What happens if we just shove Perfect Transparent Newcomb into the form of a suitable Nirvana-free $\Theta$? Neglecting some conditions like upper-completion, $\Theta \left({\pi }_{1,1}\right)$ and $\Theta \left({\pi }_{1,2}\right)$ both have the single point where the box is guaranteed-full and you 1-box for a hundred bucks. $\Theta \left({\pi }_{2,1}\right)$ has the single point where the box is guaranteed-empty and you 1-box for nothing. $\Theta \left({\pi }_{2,2}\right)$ has the single point where the box is guaranteed-empty and you 2-box for 1 dollar.

But this violates the pseudocausality condition. The outcome "the box is empty, and you 2-box" is in $\Theta \left({\pi }_{2,2}\right)$, and supported over $F^{NF}\left({\pi }_{1,2}\right)$. However, this distribution isn't present in $\Theta \left({\pi }_{1,2}\right)$. The pseudocausality condition wants us to add the bad outcome back in!

One more stab. Let's look at the non-closed set of environments (with Nirvana) corresponding to $ϵ$-noise Transparent Newcomb, where $0.01\ge ϵ>0$. The value 0.01 isn't that important, the key part is just that we are in a situation where there's some noise in the prediction and we know it's small but we don't know how small. We can try taking the closure either before or after we turn it into a $\Theta$, and see what happens.

Taking the closure in the space of a-environments, and turning it into a $\Theta$ ends up adding in the bad distribution where it's assumed we 2-box, and sadly, this is Nirvana-free for a 1-boxing strategy. So we get the same failure mode as before.

Something rather interesting happens if we take the closure of the $\Theta \left(\pi \right)$ sets after translating this problem into a Nirvana-free belief function though. For $\Theta \left({\pi }_{1,2}\right)$, we've got a bunch of distributions going "$1-ϵ$-probability on a full box and we 1-box, $ϵ$-probability on an empty box and we 2-box" Taking the closure won't add the bad distribution, it'll only add in the desired Perfect Transparent Newcomb distribution where it's full and you 1-box, guaranteed. But we still don't have pseudocausality.

This mismatch points towards needing a condition that's something like "you can't add in a Nirvana-free a-measure to $\Theta \left(\pi \right)$ if it's a limit of Nirvana-containing a-measures, you can only add it in if it's a limit of Nirvana-free a-measures". Murphy can't take arbitrary limits to make an inconvenient a-measure for you, they can only take limits of things that don't give you a shot at infinite value.

Our fundamental problem is that the expected value of $0.01\ge ϵ>0$ Newcomb, if Omega falsely has "2-box on full-box" locked in as the strategy, and we 1-box in that situation instead, goes "infinity, infinity, infinity... 1 dollar". Limits can lead outside the set of Nirvana-containing a-measures.

What would happen if we decreed that the set of Nirvana-containing a-measures was closed? The limit point of our sequence should still have infinite value, so it should be something like "probability 1 of an empty box and we 2-box, probability $0^{+}$ of a full box and we 1-box leading to Nirvana, of infinite value", where $0^{+}$ is interpreted as an infinitesimal quantity. Murphy still avoids this, because Nirvana can't truly be ruled out. Or, maybe we could slap some distance metric on a-measures that goes "this one has some probability of Nirvana here, this one doesn't, so they don't get arbitrarily close"

Definition 17: Surmeasure

A surmeasure is a pair of a measure and a function mapping Nirvana-histories to "possible" or "impossible", where Nirvana-histories with positive measure must be labeled "possible". Possible Nirvana-histories with 0 measure are assigned a value of $0^{+}$. A-surmeasures and sa-surmeasures are defined in the obvious way from this, and denoted by $SM$.

And a survironment is like an environment, except it can have $0^{+}$ chance of getting "impossible" observations and from there you're guaranteed to get Nirvana at some point.

Definition 18: Surtopology

The surtopology is the topology over the space of (s)a-(sur)measures with a subbasis of the usual open balls around points, and the sets $\left\{\right(m,b\left)|m\right(h)=0\}$ where $h$ ranges over the histories that end with Nirvana.

Definition 19: Surmetric

The surmetric is the metric over the space of (s)a-(sur)measures defined as:

$d_s\left(\right(m_1,b_1),(m_2,b_2\left)\right)=max\left(d\right(m_1,m_2)+|b_1-b_2|,{\gamma }^{t(m_1,m_2)})$

where $d$ is the usual KR-metric, $\gamma <1$, and $t$ is the minimum length of a Nirvana-containing history that is possible/positive measure according to $m_1$ and impossible/0 measure according to $m_2$ (or vice-versa)

Murphy will avoid all points with any chance of Nirvana. Remember, we only need to deal with these shenanigans if our decision-theory problem has the environment completely lock us out of being able to disprove a false prediction. Or, if you really want to deal with these decision-theoretic cases and don't want to deal with surmeasures, you could just stick with ordinary acausal hypotheses and avoid attempting to cram them into a causal hypothesis/set of environments.

Interestingly enough, these three ways of dodging the problem are equivalent!

Theorem 1: The surmetric on the space of sa-measures $M^{sa}\left(F\right({\pi }_{pa}\left)\right)$ induces the surtopology. The Cauchy completion of $M^{sa}\left(F\right({\pi }_{pa}\left)\right)$ w.r.t the surmetric is exactly the space of sa-surmeasures.

If you didn't quite follow the previous discussion, the tl;dr version of it is that the standard way of getting a set of environments from a set of policy-dependent environments, where you hardcode in all possible policies and you go to Nirvana if you violate the hardcoded policy, only works when you're guaranteed to have some chance at Nirvana if you're mispredicted. Some decision theory problems with perfect predictors violate this property, but in that case, we can still make it work by going "oh, the probability of getting into a situation where you're mispredicted isn't 0, it's just arbitrarily small". Letting in outcomes that are possible but have arbitrarily low measure, gets us survironments and surmeasures, so we can view all policy-dependent decision theory problems this way, as arising from a set of survironments (with that $0^{+}$ chance of some outcomes), instead of just a vanilla set of environments.