It seems useful to consider agents that reason in terms of an unobservable ontology, and may have uncertainty over what this ontology is. In particular, in Dialogic RL, the user's preferences are probably defined w.r.t. an ontology that is unobservable by the AI (and probably unobservable by the user too) which the AI has to learn (and the user is probably uncertain about emself). However, onotlogies are more naturally thought of as objects in a category than as elements in a set. The formalization of an "ontology" should probably be a POMDP or a suitable Bayesian network. A POMDP involves an arbitrary set of states, so it's not an element in a set, and the class of POMDPs can be naturally made into a category. Therefore, there is need for defining the notion of a probability measure over a category. Of course we can avoid this by enumerating the states, considering the set of all possible POMDPs w.r.t. this enumeration and then requiring the probability measure to be invariant w.r.t. state relabeling. However, the category theoretic point of view seems more natural, so it might be worth fleshing out.
Ordinary probably measures are defined on measurable spaces. So, first we need to define the analogue of "measurable structure" (σ-algebra) for categories. Fix a category C. Denote Meas the category of measurable spaces. A measurable structure on C is then specified by providing a Grothendick fibration B:MFC→Meas and an equivalence E:B−1(pt)→C. Here, B−1(pt) stands for the essential fiber of B over the one point space pt∈Meas. The intended interpretation of MFC is, the category of families of objects in C indexed by measurable spaces. The functor B is supposed to extract the base (index space) of the family. We impose the following conditions on MFC and B:
Given A∈Meas, Y∈MFC and f:A→B(Y), we denote the corresponding base change by fY:f−1(Y)→Y (f−1(Y)∈MFC and B(f−1(Y)) is canonically isomorphic to A).
Consider X,Y∈MFC and g,g′:X→Y. Consider also a point q∈B(X). We can think of q as a morphism q:pt→X. This allows us considering the base changes Xq:=q−1(X) and Yf(q) (the "fibers" of X at q and Y at f(q) respectively) where f:=B(g). Applying the universal property of Yf(q) to g∘qX and g′∘qX, we get morphisms gq,g′q:Xq→Yq. We now require that, if for any q∈B(X), gq=g′q then g=g′ (morphisms between families that are pointwise equal are just equal).
Consider X,Y∈MFC and g:X→Y. Suppose that (i) B(g) is an isomorphism and (ii) for any q∈B(X), gq is an isomorphism. Then, g is an isomorphism (families with a common base that are pointwise isomorphic are just isomorphic).
I'm not entirely sure how sufficient or necessary these conditions are for proving useful results, but they seem to me natural at first glance. Note that this definition can be regarded as motivated by the Yoneda lemma: a measurable space A∈Meas is defined by the measurable mappings to A from other measurable spaces, so a "measurable category" should be defined by the measurable "mappings" to it from measurable spaces, and MF is precisely the category of such measurable "mappings". Compare this with definition of geometric stacks(fn1).
Next, we define probability measures. Specifically, for any "measurable category" C (a category equipped with structure as above), we construct the category ΔC of "probability measures on C". First, we define the auxiliary category ~ΔC. An object in ~ΔC is a pair (X,μ) where X is an object in MFC and μ is a probability measure on B(X). We interpret this as sampling q∈B(X) from μ and then taking Xq (using E, the latter can be considered to be an object in C). We define the morphisms from (X,μ) to (Y,ν) as those morphisms g:X→Y for which B(g)∗μ=ν (the notation stands for pushforward). Given g:X→Y, we call it a "quasi-isomorphism" when, for any q∈B(X), gq is an isomorphism. Claim: quasi-isomorphisms admit a calculus of right fractions(fn2). We now define ΔC as the localization of ~ΔC by quasi-isomorphisms.
(fn1) Maybe the analogy with stacks should be made more formal? Not sure, stacks are motivated by topology and measurable spaces are not topological...
(fn2) This should clearly be right, and this is right for natural examples, but I haven't written down the proof. If it turns out to be false it would mean that my conditions on MFC are too weak.