In a comment on my post on topological truth predicates, Paul suggests an approach that uses probabilistic oracle machines instead, in order to make this work more comprehensible to computer scientists. I like this idea!

Paul sketches a framework developed by him and Jessica Taylor, based on a conversation with Scott Aaronson; in this post, I propose a slight simplification of their framework. My version has an oracle $O(┌M┐,p)$, which takes the source code of a probabilistic oracle machine $M$ and a $p\in Q\cap [0,1]$. If for every possible oracle $O^{\prime }$, $M\left[O^{\prime }\right]$ halts with probability one and outputs either $0$ or $1$, then $O(┌M┐,p)$: (i) returns "true" if the probability that $M\left[O\right]$ returns $1$ is $>p$; (ii) returns "false" if it is $<p$; (iii) randomly returns "true" or "false" if it is exactly $p$. If $M$ is not of this form, the oracle always returns "false". I think this is sufficient for all the applications I have in mind---in particular, for defining a variant of AIXI that is able to reason about worlds containing other, equally powerful variant AIXIs (but many details remain to be checked).

---

To be clear, by "probabilistic oracle machine" I mean a probabilistic Turing machine with access to our oracle, which itself behaves probabilisticly. Let's write $P:=Q\cap [0,1]$ for the set of all rational probabilities, and let's consider an oracle to be a function $O:N\times P\to [0,1]$, such that $O(┌M┐,p)$ specifies the probability that the oracle will return "true" when invoked on the pair $(┌M┐,p)$. (If the oracle is invoked multiple times on the same arguments, each of these is an independent random event.) The function $O(┌M┐,p)$ plays a similar role in this framework to the function $\theta (┌\phi ┐,┌\psi ┐)$ from the topological truth predicates post. Feedback on notation and terminology is encouraged---my exposure to theoretical CS is limited.

Let's define a query to be a probabilistic oracle machine $M$ such that for every oracle $O$, $M\left[O\right]$ halts with probability one and outputs either $0$ or $1$. If $M$ is a query, let $P(M,O)$ be the probability that $M\left[O\right]$ outputs $1$. Then our condition above can be re-stated as follows:

(i) If $P(M,O)>p$, then $O(┌M┐,p)=1$. (ii) If $P(M,O)<p$, then $O(┌M┐,p)=0$. (iii) If $P(M,O)=p$, then $O(┌M┐,p)\in [0,1]$ is arbitrary. (iv) For any $n$ which is not the Gödel number of a query, $O(n,p)=0$.

The last of these conditions is a bit of a devious trick; it gives the machine access to a fairly powerful halting oracle, which can e.g. check whether a probabilistic Turing machine halts with probability one. (Given such a machine $T$, construct the machine $T^{\prime }$ which runs $T$ and outputs $1$ if $T$ halts; then if $T$ halts with probability $1$, $T^{\prime }$ is a query and $O(┌T^{\prime }┐,0)=1$, but if $T$ halts with probability $<1$, $T^{\prime }$ is not a query and $O(┌T^{\prime }┐,0)=0$.) I think that I need this in my variant of AIXI in order to filter out "world models" which don't necessarily halt, and I think this will be enough to do so, but I'll leave working out the details to a later post.

At the end of this post, I'll prove that a $O$ with these properties exists. But first, let me sketch how it can be used.

---

Suppose that we have an agent which is trying to choose between two actions, $A$ and $A^{\prime }$, and suppose that the agent has two probabilistic oracle machines, $M$ and $M^{\prime }$, which represents the agent's model of what will happen in the world if it chooses action $A$ or action $A^{\prime }$, respectively. (This doesn't mean that the agent has to have perfect knowledge about how the world works---since these are probabilistic machines, they can represent probability distributions over the laws of physics.) We could think of $M$ and $M^{\prime }$ as outputting information about things that happen in the world, which we can then plug into a computable utility function, but it will be easier to think of them as directly returning a utility.

Our goal, then, is for our agent to choose $A$ if the expectation of the utility returned by $M$ is greater than the expectation of the utility returned by $M^{\prime }$.

Utilities are real numbers (and if we consider infinite-horizon problems with temporal discounting, rational utilities won't suffice), so we need a way for probabilistic machines in our framework to represent probability distributions over reals. Let's say that an machine "outputs the real number $r\in [0,1]$" if it outputs a sequence of nested closed intervals with endpoints in $P=Q\cap [0,1]$, converging to the single point $r$. We'll assume that $M\left[O^{\prime }\right]$ and $M^{\prime }\left[O^{\prime }\right]$ output real numbers in $[0,1]$ with probability one, for every oracle $O^{\prime }:N\times P\to [0,1]$. I'll write $E\left[N\right]$ for the expected value of the number returned by $N\left[O\right]$, if $N$ is such an oracle machine (leaving the oracle $O$ implicit in the notation).

We want to find an oracle machine that will output $A$ if $E\left[M\right]>E\left[M^{\prime }\right]$, output $A^{\prime }$ if $E\left[M\right]<E\left[M^{\prime }\right]$, and output either $A$ or $A^{\prime }$ if the expectations are equal. Note that in our representation, we can write machines that compute e.g. the sum or difference of the numbers computed by two other machines; and note that $E\left[M\right]>E\left[M^{\prime }\right]$ is equivalent to $E\left[\right(1+M-M^{\prime }\left)/2\right]>0.5$. We have $E\left[\right(1+M-M^{\prime }\left)/2\right]\in [0,1]$; thus, if we could find a query $N$ such that $N\left[O\right]$ returns $1$ with probability $E\left[\right(1+M-M^{\prime }\left)/2\right]$, we could simply call our oracle on $O(┌N┐,0.5)$, and take action $A$ if the oracle returns "true", action $A^{\prime }$ otherwise.

We can implement $N$ as follows. Set $x_0:=0$ and $y_0:=1$, and initialize a run of $(1+M-M^{\prime })/2$. Then, for every $n\in N_{\ge 1}$: run $(1+M-M^{\prime })/2$ until it returns its next ($n$'th) interval, with lower bound $x_n$ and upper bound $y_n$. Then, with probability $(x_n-x_{n-1})/(y_{n-1}-x_{n-1})$, halt and output $1$; with probability $(y_{n-1}-y_n)/(y_{n-1}-x_{n-1})$, halt and output $0$; otherwise, continue looping.

Since with probability one, $(1+M-M^{\prime })/2$ outputs a sequence of intervals that converge to a single point, $N$ halts with probability one and outputs either $0$ or $1$, whatever oracle we run it on; i.e., it is a query. We can describe its behavior as sampling a random number $r\in [0,1]$ from the distribution $(1+M-M^{\prime })/2$, and then outputting $1$ with probability $r$ and $0$ with probability $(1-r)$. When looked at it this way, it's clear that when run on $O$, this machine will output $1$ with probability $E\left[\right(1+M-M^{\prime }\left)/2\right]$, as desired.

---

Let's now proceed to the proof that an appropriate $O$ exists. This is analogous to the proof in the topological truth predicates post. (However, while we'll still need to use the product topology and the infinite-dimensional generalization of Kakutani’s fixed point theorem, this proof won't require knowledge of Grothendieck universes, set theory, or formal logic.)

Recall that an oracle is an arbitrary function $O:N\times P\to [0,1]$. We say that an oracle $O^{\prime }$ reflects another oracle $O$ if for every $p\in P$ and every query $M$, we have $O^{\prime }(┌M┐,p)=1$ if $P(M,O)>p$ and $O^{\prime }(┌M┐,p)=0$ if $P(M,O)<p$, and for any $n$ that is not the Gödel number of a query, we have $O(n,p)=0$. In interpretation, $O^{\prime }$ reflects $O$ if it gives correct information about the probability that a query returns $1$ when it's executed with oracle $O$.

An oracle is called reflective if it reflects itself; reflective oracles give answers which are true about the oracle itself.

Our main goal in this post is just to show that reflective oracles exist. However, later I'll want to show that for every Nash equilibrium of a finite game, there is a reflective oracle which makes the players of that finite game play that Nash equilibrium, and in order to do so, it will be helpful to have a slightly stronger statement, which shows that there are reflective oracles satisfying certain constraints.

These constraints will take the form of a partial function $\pi :N\times P\nrightarrow [0,1]$, which specifies the values $O$ should take on a certain set $dom\left(\pi \right)\subseteq N\times P$. We call a function of this type a partial oracle, and say that $O$ extends $\pi$ if $O(n,p)=\pi (n,p)$ for all $(n,p)\in dom\left(\pi \right)$. We call a partial oracle $\pi$ reflective if for every oracle $O$ extending $\pi$, there is an oracle $O^{\prime }$ which reflects $O$ and which also extends $\pi$.

---

With these preliminaries, we can state our existence result for reflective oracles:

Theorem.

(i) There is a reflective oracle $O:N\times P\to [0,1]$. ​ (ii) For every reflective partial oracle $\pi :N\times P\nrightarrow [0,1]$, there is a reflective oracle $O$ which extends $\pi$.

Proof. We begin by showing that the empty partial oracle ${\pi }_{\varnothing }$ (i.e., the partial oracle satisfying $dom\left({\pi }_{\varnothing }\right)=\varnothing$) is reflective, since (i) then follows from (ii). Thus, let $O$ be any oracle (since every oracle extends ${\pi }_{\varnothing }$), and define $O^{\prime }:N\times P\to [0,1]$ as follows: For any $M$ and $p$ such $M$ is a query and $P(M,O)>p$, set $O^{\prime }(┌M┐,p)=1$; for all other pairs $(n,p)$, set $O(n,p)=0$. Then, $O^{\prime }$ clearly both reflects $O$ and (trivially) extends ${\pi }_{\varnothing }$. This finishes the proof that ${\pi }_{\varnothing }$ is a reflective partial oracle.

It remains to show (ii). For this, we use the infinite-dimensional generalization of Kakutani’s fixed point theorem:

Suppose that $A$ is a non-empty, convex and compact subset of a locally convex topological vector space. Suppose further that $f:A\to Pow\left(A\right)$ is a set-valued function such that $f\left(x\right)$ is non-empty, convex and compact for all $x\in A$, and such that the graph of $f$, $$\left\{\right(x,y):x\in A,y\in f(x\left)\right\},$$ is a closed set. Then $f$ has a fixed point; that is, there is an $x\in A$ such that $x\in f\left(x\right)$.

We let $A$ be the set of all oracles extending $\pi$; then $A\subset R^{N\times P}$, which is a locally convex topological vector space when endowed with the product topology (since this is true of any power of $R$). $A$ is clearly non-empty, convex, and closed, and it is a subset of $[0,1{]}^{N\times P}$ which, by Tychonoff’s theorem, is compact. Moreover, $N\times P$ is countable, so it's sufficient to consider convergence of sequences in $R^{N\times P}$.

We choose $f\left(O\right)$ to consist of all oracles $O^{\prime }$ that reflect $O$ and extend $\pi$. By the assumption that $\pi$ is reflective, $f\left(O\right)$ is non-empty for every $O\in A$. Moreover, it is easy to see that $f\left(O\right)$ is both closed and convex; hence, if we can also show that $f$ has closed graph, then by the fixed point theorem, there is a $O\in A$ such that $O\in f\left(O\right)$, which is exactly what we want to show.

Thus, assume that we have sequences $O_n$ and $O_n^{\prime }$ of oracles extending $\pi$ such that $O_n^{\prime }\in f\left(O_n\right)$ for every $n\in N$, and suppose that these sequences converge to limits $O,O^{\prime }\in A$; then we need to show that $O^{\prime }\in f\left(O\right)$, i.e., that $O^{\prime }$ reflects $O$. To see this, we must show that for every query $M$ and any $p\in P$, $P(M,O)>p$ implies $O^{\prime }(┌M┐,p)=1$, and $P(M,O)<p$ implies $O^{\prime }(┌M┐,p)=0$. (The condition that $O^{\prime }(n,p)=0$ for any $n$ that is not the Gödel number of a query follows directly from the fact that this is true about all $O_n^{\prime }$.)

Without loss of generality, assume that $P(M,O)<p$. Let $\epsilon :=(p-P(M,O\left)\right)/2$. Since $M$ is a query, $M\left[O\right]$ halts with probability one (and returns either 0 or 1); hence, there is some $T\in N$ such that with probability $>1-\epsilon$, $M\left[O\right]$ halts in at most $T$ steps. Despite the fact that $M$ is probabilistic and can make calls to an oracle, there are only finitely many possible execution traces of $M$ during the first $T$ timesteps, and thus only finitely many pairs $(┌M^{\prime }┐,p^{\prime })$ on which $M$ might invoke its oracle during these timesteps.

Since $O_n$ converges to $O$, there is an $n_0$ such that for all $n\ge n_0$, $\left|O_n\right(┌M^{\prime }┐,p^{\prime })-O(┌M^{\prime }┐,p^{\prime }\left)\right|<\epsilon /T$ for each of these finitely many pairs $(┌M^{\prime }┐,p^{\prime })$. Suppose that we model each call to the oracle as sampling a random number from a uniform $[0,1]$ distribution, and returning "true" if this number is greater than $O(┌M^{\prime }┐,p^{\prime })$; then the above result implies that with probability $>1-\epsilon$, $M\left[O_n\right]$ and $M\left[O\right]$ behave the same way during the first $T$ timesteps. Hence, with probability $>1-2\epsilon$, $M\left[O_n\right]$ and $M\left[O\right]$ both halt during the first $T$ steps and return the same result. But this implies that $P(M,O_n)<P(M,O)+2\epsilon =p$, for all $n\ge n_0$.

Since $O_n^{\prime }$ reflects $O_n$, this implies $O_n^{\prime }(┌M┐,p)=0$, and since this holds for all $n\ge n_0$ and $O_n^{\prime }$ converges to $O^{\prime }$, it follows that $O^{\prime }(┌M┐,p)=0$, as desired. □