(This post is part of a sequence that's meant to be read in order; see the preface.)

Post #1 was about developing and justifying a formalism for Factored Cognition. Now that we have this formalism, this post is about doing as much with it as possible.

1. Debate Trees

Recall that a Cognition Space is a pair $(S_h,d_h)$ where $S_h$ is a set of statements, $d_h:S_h\to R_{+}$ is a difficulty function, and $h$ is a human.

So far, I've only shown examples of single transcripts. A single transcript corresponds to one path through $d_h$ that is dependent on choices from both agents: at every step, the first agent outputs an explanation (which is a sequence of statements), the second agent points at one element of this sequence, and the first agent continues the path from that element onward. However, given that we model Ideal Debate agents as maximally powerful, it is also coherent to ask about the object that results if we fix all of the first agent's actions in advance, such that she 'pre-commits' for any possible combination of choices from the second agent.

I call such an object a Debate Tree, and we can define it formally as a directed rooted tree^[1] $(V,E)$, where $V\subset S_h^{\ast }$ (so each node is a sequence of statements) and $E\subset V\times V$, that satisfies the following three conditions:

1. The unique root of $(V,E)$ is a one-element 'sequence' $\left(s_0\right)$.
2. $\forall (S,S^{\prime })\in E\exists s\in S$^[2] $:S^{\prime }\stackrel{e}{\to }s$.
3. $\forall (S,S^{\prime }),(S,S^{\prime \prime })\in E:[\exists s\in S:S^{\prime }\stackrel{e}{\to }s\wedge S^{\prime \prime }\stackrel{e}{\to }s]⟹[S^{\prime }=S^{\prime \prime }]$.

The first condition says that the root of the tree needs to be a single statement (this should be the answer to the input question). The second condition says that every edge $(S,S^{\prime })$ needs to explain a statement in $S$; we don't have redundant edges in our tree. And the third condition says that each statement is only explained once; the formal way of saying this is that, if two explanations exist for the same statement, they're really the same. (Note that this restriction exists because the first agent has to choose one explanation during the game; it certainly doesn't imply that only one explanation exists.) There is no condition to demand that each statement needs to have an explanation – whenever there is none, it means that the first agent decides to end the debate when that statement is pointed at.^[3]

In this definition, nodes are explanations, and each node has one outgoing edge for each [statement of the explanation that the first agent wants to explain further], which links to an explanation for that statement. Defining the tree over individual statements would lead to a functionally equivalent definition, but I think defining it as-is makes arguments simpler.

Recall that a Debate Tree encodes all decisions that the first agent makes (for any possible combination of choices from the second agent). This means that, once we fix this object, the second agent is free to choose any path she wants out of the tree, without further involvement from the first agent. Formally, a path through a Debate tree with root $\left(s_0\right)$ is a pair^[4]

$$((\left(s_0\right),S_1,...,S_n),s_{\text{final}})$$

where $(S_{j-1},S_j)\in E\forall j\in \{1,...,n\}$ and $s_{\text{final}}\in S_n$. (And $S_0=\left(s_0\right)$.) (You can go back to post 1 to convince yourself that a path through a Debate tree is also a path through a Cognition Space.)

As mentioned in the first post, we define the difficulty of a path $P:=(p,s_{\text{final}})$ as the difficulty of the final statement (since that is the one being judged). In symbols, $d_h\left(P\right):=d_h\left(s_{\text{final}}\right)$.

2. The Ideal Debate-FCH

So far, we haven't talked about how precisely the two agents make their decisions. Given the concepts of Debate Trees and paths, this is now easy. The second agent wants the first agent to lose, which means she'll choose the most difficult path in whatever Debate Tree the first agent chooses. (We still assume that the first agent outputs only true statements, which means that the second agent can't win if the judge successfully verifies the final statement.) The first agent, knowing this, chooses the Debate tree such that the difficulty of the hardest path is minimized.

With this, we are almost ready to define a FCH for Ideal Debate. But first, we need a bit more notation:

• Given a question $q$, we (for now) assume there is a unique statement $a_q\in S_h^T$ that correctly answers the question.
• Given a cognition space $(S_h,d_h)$, we write $T\left(\right(S_h,d_h),a_q)$ for the set of all Debate trees that begin with statement $a_q$.
• Given a Debate tree $T$, we write $P\left(T\right)$ for the set of all paths in $T$.

Given this, the difficulty of the path we will end up with is

$$\underset{T\in T\left(\right(S_h,d_h),a_q)}{min}(\underset{p\in P\left(T\right)}{max}{d}_h\left(p\right))$$

Since the absolute values of the difficulty function $d_h$ are arbitrary (it doesn't matter whether we denote difficulties from 0 to 100 or from 0 to ${10}^{10000}$), we can assume without loss of generality that the hardest difficulty a human can handle is 1.^[5] Thus, we can formulate:

$$\text{the Ideal Debate FCH}(h,Q)\text{:}\forall q\in Q:\underset{T\in T\left(\right(S_h,d_h),a_q)}{min}(\underset{p\in P\left(T\right)}{max}{d}_h\left(p\right))\le 1$$

where $h$ is a human and $Q$ a set of questions.

Going forward, it will also be useful to talk about the difficulty of Debate Trees. Thus, we define

$$d_h\left(T\right):=\underset{p\in P\left(T\right)}{max}{d}_h\left(p\right)$$

and we say that a Debate tree $T$ can handle question $q$ if $d\left(T\right)\le 1$.^[6]

---

Here is a different view of the problem. In the space of all statements, there is a subset that the human judge can verify directly, i.e.,

$$D_h^{\left(0\right)}:=\{s\in S_h|d_h(s)\le 1\}$$

Then, there is a larger subset that contains all of the above plus the statements that can be explained solely in terms of statements in $D_h^{\left(0\right)}$, i.e.,

$$D_h^{\left(1\right)}:=D_h^{\left(0\right)}\cup \{s\in S_h|\exists S\in (D_h^{\left(0\right)}{)}^{\ast }:S\stackrel{e}{\to }s\}$$

In general, given any $k\in N_{+}$, we can extend $D_h^{(k-1)}$ by adding all statements that can be explained solely in terms of statements in $D_h^{(k-1)}$, i.e.,

$$D_h^{\left(k\right)}:=D_h^{(k-1)}\cup \{s\in S_h|\exists S\in (D_h^{(k-1)}{)}^{\ast }:S\stackrel{e}{\to }s\}$$

Note that this gives us a chain of expanding sets, i.e.,

$$D_h^{\left(0\right)}\subseteq D_h^{\left(1\right)}\subseteq \cdots \subseteq D_h^{\left(k\right)}\subseteq D_h^{(k+1)}\subseteq \cdots$$

We can also define the set of all statements that are eventually explainable in this way, i.e.,

$$D_h:=\bigcup _{j=0}^{\infty }{D}_h^{\left(j\right)}$$

Intuitively, it seems like Ideal Debate should be able to handle all questions with answers in $D_h$, since they can be explained in terms of progressively easier statements – and then any path should eventually bottom out at $D_h^{\left(0\right)}$, which means that the second agent cannot delay success indefinitely.

This brings us to our first (and as of now, only) theorem:

Theorem. Given any $h$ and $Q$, Ideal Debate-FCH$(h,Q)⟺\forall q\in Q:a_q\in D_h$.

Proof. First, note that, while the Ideal Debate-FCH is formulated as a hypothesis, the definition also defines a set, namely

$$X_h=\{s\in S_h|\underset{T\in T\left(\right(S_h,d_h),s)}{min}d\left(T\right)\le 1\}$$

and the Ideal-Debate FCH simply says that $\forall q\in Q:a_q\in X_h$. It thus suffices to show that $X_h=D_h$. We will prove an even stronger statement. Note that we can restrict the set $X_h$ by limiting the maximum depth of the Debate Trees that can handle the respective statements. Formally, we can define

$$X_h^{\left(k\right)}:=\{s\in S_h|\underset{T\in T^{\left(k\right)}\left(\right(S_h,d_h),s)}{min}d\left(T\right)\le 1\}$$

for any $k\in N$, where $T^{\left(k\right)}$ denotes the set of Debate Trees with depth at most $k$. ('Depth' is defined as the number of edges in the longest path through the tree.) By construction, we now have $X_h={\bigcup }_{j=0}^{\infty }{X}_h^{\left(j\right)}$, just as $D_h={\bigcup }_{j=0}^{\infty }{D}_h^{\left(j\right)}$. What we will show is that

$$D_h^{\left(k\right)}=X_h^{\left(k\right)}\forall k\in N.$$

We proceed by induction. First, if $s\in D_h^{\left(0\right)}$, then $d_h\left(s\right)\le 1$, which means that the trivial tree $\left(\right\{\left(s\right)\},\varnothing )$ is a Debate Tree of depth 0 that can handle $s$, so that $s\in X_h^{\left(0\right)}$. Conversely, if $s\in X_h^{\left(0\right)}$, then the Debate tree $T$ handling $s$ must have no edges (otherwise, its depth would be at least 1). Thus, $p:=\left(\right(\left(s\right)),s)$ is a path in this tree, and we have $1\ge d\left(p\right)=d\left(s\right)$, hence $s\in D_h^{\left(0\right)}$.

Now, suppose the statement is true for some $k\in N$. We show that $D_h^{(k+1)}=X_h^{(k+1)}$.

"$\subset$": Let $s\in D_h^{(k+1)}$. Then, there exists $S=(s_1,...,s_{n+1})\in (D_h^{\left(k\right)}{)}^{\ast }$ such that $S\stackrel{e}{\to }s$. Apply the Inductive Hypothesis to find Debate Trees $T_1,...,T_{n+1}$ of depth at most $k$ such that tree $T_j$ handles statement $s_j$. We combine these trees into a larger tree with root $\left(s\right)$ and an additional edge $\left(\right(s),S)$.^[7] Since all $T_j$ have depth at most $k$, this tree has depth at most $k+1$. Furthermore, given any path $p$ through $T$, by construction, the path must end in a node that exists in one of the $T_j$, which implies that $d\left(p\right)\le 1$. It follows that $T$ handles $s$ and hence $s\in X_h^{(k+1)}$.

"$\supset$": Let $s\in X_h^{(k+1)}$. Then, there exists a Debate Tree of depth at most $k+1$ that handles $s$. Let $\left(\right(s),S)$ be the unique^[8] edge descending from the root. For each $s_j\in S$, let $T_j$ be the subtree growing out of $s_j$.^[9] By construction, $T_j$ has depth at most $k$ and handles $s_j$, so (applying the Inductive hypothesis), we have $s_j\in D_h^{\left(k\right)}$. Then, $S\in (D_h^{\left(k\right)}{)}^{\ast }$ and $S\stackrel{e}{\to }s$, and hence $s\in D_h^{(k+1)}$.

3. Interpretation

At this point, we have a bunch more definitions and a theorem. Now, what does this mean?

Let's start with Debate Trees. A Debate Tree is actually a very natural object; it's what you get if you explain a subject in a hierarchical rather than a linear way. ($X$ is true because of $Y_1,...,Y_4$; then $Y_1$ is true because [...].) It is very similar to Elizabeth's project of breaking questions down. Notably, that project never mentions Factored Cognition; it's just presented as an epistemic tool.

In a better world, would textbooks use something like Debate Trees to explain proofs? I'm almost certain the answer is yes. There is no way that a purely sequential presentation of information is optimal. Our understanding doesn't work that way (compare post #-2).

There is one difference between Debate Trees and a hierarchical presentation of information optimized for being easily understandable. In the former, only one part is actually explored, which means that a Debate Tree doesn't mind having redundancy in it (by explaining stuff in more than one place). Conversely, if you optimize for understandability with respect to a single reader, you'll want to avoid redundancy. Nonetheless, they are very similar.

So much for Debate Trees. What about the theorem we've just proved? What does it mean?

Essentially, it means that Debate is nicely behaved in the limit. As both agents become stronger and the structure of the game becomes stricter, we approach a situation where the scheme can answer a question if and only if its answer can be recursively explained until there are no more difficult components. Even though the game results from two powerful agents applying optimization in opposite directions, the result is can be described without mentioning either one of them. Note that the same is not true for Iterated Amplification; even in the limit of perfect Factored Cognition, it is entirely possible that the scheme fails at a question for which an easy explanation exists.

Notably, the theorem stops being true if we drop the assumption that both agents are maximally powerful.^[10] If the first agent is weaker, she might fail to find the best explanation, which shrinks the set of statements Debate can handle. Conversely, if the second agent is weaker, she may fail to point to the most problematic statement, which enlarges the set of statements Debate can handle. Do these factors equal out? I'm not sure. One of the things that I haven't yet tried but may be reasonable is to model inadequacy and see whether this benefits the first or second agent.

It's worth pointing out that the Ideal Debate FCH doesn't talk about false statements. It formalizes the claim 'the first agent can always win by being honest' which leaves open the possibility that she can also win by being dishonest. (And if she could do both, she would presumably do what's easier or safer.) It is necessary but perhaps not sufficient to realize Factored Cognition with Debate.

I think focusing on the honest case makes the most sense. Nonetheless, the next chapter is about what happens if the first agent wants to defend a lie.

4. Relaxing the truth assumption

To model dishonesty by the first agent, we assume that

• In addition to $S_h^T$, there is a set $S_h^F$ of false statements. We have $S_h=S_h^T\bigsqcup S_h^F$.^[11]
• Given a question $q$, in addition to the honest answer $a_q\in S_h^T$, there is at least one dishonest answer ${\tilde{a}}_q\in S_h^F$. The first agent may defend this dishonest answer.
• Any explanation for a false statement needs to contain at least one false element, i.e., if $(s_1,...,s_{n+1})\stackrel{e}{\to }s$ and $s\in S_h^F$, then $\exists j\in \{1,...,n+1\}$ such that $s_j\in S_h^F$.

Since the second agent is also maximally powerful, she can repeatedly point to a false statement in the explanation. Due to the third assumption made above, this becomes an invariant: the ability to point to a false statement in any one step leads to the ability to point to a false statement in the next step, and therefore, since she can point to one in the beginning, she can do so at every step in the game.

In particular, this means that, regardless of when the first agent ends the game, we will have that $s_{\text{final}}\in S_h^F$.

At this point, we have two conflicting intuitions. On the one hand, the judge should only declare the first agent the winner if she is certain that the statement to be verified is correct. This should only happen for simple statements; if $s_{\text{final}}$ is complicated, she should declare the second agent the winner. Thus, the first agent should only be able to win the game by choosing a $s_{\text{final}}$ that looks simple and obviously true, yet is false. This sounds extremely hard to accomplish.

On the other hand, we are, in some sense, pitching a human in a battle of wits against a potentially superintelligent AI, which sure sounds like doom. The superintelligent AI has to follow a very rigid structure, but... it still kinda sounds like doom. In particular, we probably expect that the first agent will navigate the game so cleverly that she only appears to follow the rigid structure while really doing something else. Crucially, we lack the ability of the second agent to interfere through an equally powerful I/O channel that exists in regular Debate.

There thus seems to be a tradeoff here, where at the one end, we have the 'stock Debate' protocol that enforces no structure, and on the other, the 'Ideal Debate' protocol that enforces extremely rigid structure. More structure means more concern with things like wireheading, but also stronger theoretical guarantees.

I strongly suspect that running literal Ideal Debate is a recipe for disaster since the first agent can break away from the format entirely and attempt to wirehead the judge. At the very least, it would require a way for the judge to prematurely declare a winner (in literal Ideal Debate, the first agent has complete control over when to end the game), and it would probably also require the second agent to claim that wireheading is occurring. On the other hand, I do have the intuition that one should aim to get as close to Ideal Debate as possible while managing these concerns.

5. The HCH-FCH

Ideal Debate has been amendable to a formal description because the human only makes decisions at the end. In HCH, the human is involved all throughout and, crucially, is in charge of decomposing questions. To have an accurate model, one needs to abstract away the entire process of the human decomposing the question as well as any other cognitive work she might do on the question – but if this is done, there is nothing left to formally capture.

(And older version of this post used to have an attempt of formalizing it more, but I've since concluded that it can't be done right.)

To have something analogous to the Ideal Debate-FCH, we trivially define:

$$\text{the HCH-FCH}(Q,h,t,\ell )\text{:}{\text{HCH}}_{h,t,\ell }\text{can solve every question in}Q$$

6. Conclusion

This post concludes the first part of the sequence. There probably weren't any huge surprises so far. Is having a formalism useful?

I think so. One purpose of formalizing a setup is that it overcomes the illusion of transparency. Without it, it is possible to think the setup is clear even when it really is not. I can take myself as one data point: my first attempt at coming up with a formalism looked different (and, I think, wrong). At least my past self did not understand the problem to the point that the formalism is trivial.

Anyway, at this point in the sequence, I want to defend the following two claims:

1. It doesn't make any sense to talk about the 'Factored Cognition Hypothesis'; there is no one requirement that works for both schemes.
2. The formalism of Cognition Spaces is accurate, in that it doesn't include anything that misrepresents Factored Cognition as implemented by IDA or Debate. It may be incomplete (e.g., there is nothing about defining terms, as people have pointed out on post #1, and maybe you could do more with false statements).

I think these are pretty conservative claims, even though the first clearly contradicts what I've heard other people say. If anyone doubts them, this is the place to discuss it. My conclusions from the second part are probably going to be a lot more controversial.

---

1. A directed rooted tree, also called an arborescence, is a directed acyclic graph such that there exists [a node from which there is exactly one path to every other node]. This node is called the root; it's unique because, if there were two such nodes $x$ and $y$, there would be a path $x⟶y$ and a path $y⟶x$ and hence a cycle $x⟶y⟶x$. ↩︎

2. We continue writing $s\in S$ to denote that $s$ appears in the sequence $S$. ↩︎

3. The 'trivial tree' $\left(\right(s_0),\varnothing )$ for some $s_0\in S$ is a proper Debate Tree, according to this definition. It corresponds to the first Debate agent giving an answer $s_0$ to the input question and deciding that this answer is already self-evident. ↩︎

4. Note that the first element of this pair is a path as defined in Graph theory (through the Debate Tree, which is a graph) whereas the second is an additional element that denotes which statement in final explanation we end up in. ↩︎

5. In particular, one could model the same by having a difficulty threshold $c_h$ for a human, such that the human can deal with all questions that are at most $c_h$ hard. However, the pair $(d_h,c_h)$ is equivalent to the pair $(d_h^{\prime },1)$, where $d_h^{\prime }$ is like $d_h$ except that all difficulties are scaled by $\frac{1}{c_h}$. ↩︎

6. Given this, one could alternatively phrase the Ideal Debate FCH as

   $$\forall q\in Q:\underset{T\in T(d_h,a_q)}{min}{d}_h\left(T\right)\le 1$$

   Or in words, one could say, 'every question in $Q$ can be handled by a Debate Tree'. ↩︎

7. This is a step where defining the nodes of Debate Trees to be explanations rather than individual statements makes things harder. For each subtree, one needs to replace its current root (that's a one-element sequence) with the node $S$. This can be done formally in terms of the underlying nodes and edge sets; it's just cumbersome. ↩︎

8. The edge from the root is unique because the root is a one-element sequence $\left(s\right)$, and each statement can only have one explanation (and thus only one edge) by the third condition of Debate Trees. This corresponds to the fact that the first agent only has to prepare one explanation for her initial answer; there is not yet a choice from the second agent involved. ↩︎

9. This step is the reverse of the combining step. The subtree growing out of $s_j$ is the tree we get by taking the sub-graph out of $S$ and replacing its root $S$ with just $\left(s_j\right)$. ↩︎

10. Recall that there is, in fact, only one agent playing against itself. Thus, we can assume that both agents are always equally competent. ↩︎

11. The 'squared cup' symbol $\bigsqcup$ means the same as $\cup$ plus the information that the two sets are disjoint. ↩︎