In my previous $post$ I talked about read-eval-print loops as providing a type signature for agents. I will now explain how you can quickly transition from this framework to an ELK solution. Notation is imported from that post.

Imagine we have two agents, a human and a strong AI, denoted H and M respectively. They both interact with the environment in lockstep, according to the following diagram.

We have the human's utility function $U_H:S_H\to Q$, which is defined on the human's model of reality. We would like to lift $U_H$ to a version $U_M:S_M\to Q$ that the machine can use to influence the world in way that is agreeable to the human, which we can do by learning a mapping F : $S_M\to S_H$ and deriving $U_M=F\circ U_H$.

But we haven't yet said what properties we want the ontology map F to have. I want to call two concepts $s_h$ and $s_m$ equal if they act the same with respect to transformation: $\forall$f. f($s_h$) = f($s_m$) → $s_h$ = $s_m$. The issue is that since the concepts have different types we cannot feed them as arguments to the same function. So instead let's say that $\forall$s:S, $Eva{l}_H$($s_h$, $Rea{d}_H$(s)) = $Eva{l}_M$($s_m$, $Rea{d}_M$(s)) → $s_h$ = $s_m$. But now we are back to the same problem where we are trying to compare concepts in two different ontologies. But this does give us a kind of inductive step where we can transfer evidence of equality between concept pairs ($s_h$, $s_m$) and ($s_h$', $s_m$'). I also believe that this kind of a coherence argument is the best we can do, since we are not allowed to peer into the semantic content of particular machine or human states when constructing the ontology map.

Consider the following two graphs.

My intuition is that even if I don't know the labels of the above graphs, I can still infer that the bottom nodes correspond to each other. And the arrows that I get in the context of ELK are the agents' Eval transitions, leading to the following commutative diagram specification for F.

We can learn an ontology map F : $S_M\to S_H$ by minimizing the difference between two paths from a state $s_m$, one in which the machine's prediction function is used and one in which the human's prediction function is used.  Concretely, I propose minimizing Dist($s_{h_1}$,$s_{h_2}$) + $\lambda$ |U($s_{h_1}$)-U($s_{h_2}$)| where $s_{h_1}$ = F($Eva{l}_M$($s_m$,$o_m$)) and $s_{h_2}$ = $Eva{l}_H$(F($s_m$),$o_h$), Dist is a distance metric in $S_H$, and observations $o_m$ and $o_h$ are generated by the same underlying state S.

If you are interested in getting more detail and why I believe this circumvents existing counterexamples, please check out the full $proposal$.