Proofs are in this link

This will be a fairly important post. Not one of those obscure result-packed posts, but something a bit more fundamental that I hope to refer back to many times in the future. It's at least worth your time to read this first section up to its last paragraph.

There are quite a few places where randomization would help in designing an agent. Maybe we want to find an interpolation between an agent picking the best result, and an agent mimicking the distribution over what a human would do. Maybe we want the agent to do some random exploration in an environment. Maybe we want an agent to randomize amongst promising plans instead of committing fully to the plan it thinks is the best.

However, all of these run into the standard objection that any behavior like this, where a randomized action is the best thing to do, is unstable as the agent gets smarter and has the ability to rewrite itself. If an agent is randomizing to sometimes take actions that aren' t optimal according to its utility function, then there will be an incentive for the agent to self-modify to eliminate its randomization into those suboptimal actions.

The formalization of this is the following proposition.

Proposition 1: Given some compact metric space of options $X$, if $U:X\to R$ is a bounded function, $\{\mu |\forall \nu \in \Delta X:\mu (U)\ge \nu (U\left)\right\}=\Delta \{x|\forall y\in X:U(x)\ge U(y\left)\right\}$

Intuitively, what this is saying is that the only possible way for a mixture of options to be an optimal move is if each component option is an optimal move. So, utility functions can only give you randomization behavior if the randomization is between optimal actions. The set of such will typically only contain a single point. And so, in general, for any utility function at all, an agent using it will experience a convergent pressure towards deterministic decision-making.

Every single clever alignment trick involving an agent behaving randomly or sampling from a distribution is thereby guaranteed to fail, as it's not stable under reflection as the agent gets smarter, for anything worthy of being called an agent (in the sense that it has an implicit utility function and acts to achieve it).

The rest of this post will be about how the above sentence is false. There's a mathematically principled, reflectively stable, way an agent can be, where randomization behavior persists. No matter how smart it gets, it won't want to remove its randomization behavior. Reflectively stable quantilizers are back on the menu, as are reflectively stable human-imitators, reflectively stable Thompson samplers, and more.

What's an Infrafunction?

Intuitively, just as infradistributions are a generalization of probability distributions, infrafunctions are a generalization of functions/random variables in the same direction. The next paragraph will be informal (and somewhat wrong) to not clog it with caveats.

The Fundamental Theorem of Inframeasures says that there are two ways of viewing inframeasures. The first way is to view an inframeasure as a closed convex set of measures, where the worst-case measure is picked. You don't know what distribution in the set will be picked by reality, and so you model it as an adversarial process and plan for the worst-case. As for the second way to view an inframeasure, the thing you do with probability distributions is to take expectations of functions with them. For instance, the probability of an event is just the expected value of the function that's 1 if the event happens, and 0 otherwise. So an inframeasure may also be viewed as a functional that takes a function $X\to R$ as an input, and outputs the expectation value, and which must fulfill some weak additional properties like concavity. Measures fulfill the much stronger property of inducing a linear function $(X\to R)\to R$.

Moving away from that, it's important to note that the vector space of continuous functions $X\to R$, and the vector space of (finite signed) measures on $X$ (denoted $M^{\pm }\left(X\right)$), are dual to each other. A function $f$ and a measure $m$ are combined to get an expectation value. $f,m\mapsto \int fdm$ (ie, taking the expectation) is the special function of type $(X\to R)\times M^{\pm }\left(X\right)\to R$. Every continuous linear function $(X\to R)\to R$ corresponds to taking expectations with respect to some finite signed measure, and every continuous linear function $M^{\pm }\left(X\right)\to R$ corresponds to taking expectations with respect to some continuous function $X\to R$.

Since the situation is so symmetric, what happens if we just take all the mathematical machinery of the Fundamental Theorem of Inframeasures, but swap measures and functions around? Well... compare the next paragraph against the earlier paragraph about the Fundamental Theorem of Inframeasures.

The Fundamental Theorem of Infrafunctions says that there are two ways of viewing infrafunctions. The first way is to view an infrafunction as a closed convex set of functions $X\to R$, where the worst-case function is picked. You don't know what function in the set will be picked by reality, and so you model it as an adversarial process and plan for the worst-case. As for the second way to view an infrafunction, a thing you do with functions to $R$ is you combine them with a probability distribution to get an expectation value. So an infrafunction may also be viewed as a function that takes a probability distribution as an input, and outputs the expectation value, and which must fulfill some weak additional properties like concavity. Functions fulfill the much stronger property of inducing a linear function $M^{\pm }\left(X\right)\to R$.

For the following theorem, a set $S$ of functions $X\to R$ is called upper-complete if, whenever $f\in S$ and $g\ge f$, $g\in S$ as well. And a function $g$ will be called minimal in $S$ if $f\le g$ and $f\in S$ implies that $f=g$.

Theorem 1: Fundamental Theorem of Infrafunctions If $X$ is a compact metric space, there is a bijection between concave upper-semicontinuous functions of type $\Delta X\to R\cup \{-\infty \}$, and closed convex upper-complete sets of continuous functions $X\to R$.

Conjecture 1: Continuity=Compactness If $X$ is a compact metric space, there is a bijection between concave continuous functions of type $\Delta X\to R$, and closed convex upper-complete sets of continuous functions $X\to R$ where the subset of minimal functions has compact closure.

Ok, so, infrafunctions can alternately be viewed as concave (hill-like) functions $\Delta X\to R$, or closed convex upwards-complete sets of continuous functions.

Effectively, this is saying that any concave scoring function on the space of probability distributions (like the negative KL-divergence) can equivalently be viewed as a worst-case process that adversarially selects functions. For any way of scoring distributions over "what to do" where randomization ends up being the highest-scoring option, the scoring function on $\Delta X$ is probably going to curve down (be concave), and so it'll implicitly be optimizing for the worst-case score amongst a set of functions $X\to R$.

Now, going from a desired randomization behavior, to a concave scoring function with that desired behavior as the optimum, to the set of functions the scoring function is implicitly worst-casing over, takes nontrivial mathematical effort. There's a whole bunch of possible randomization behaviors for which I don't know what sort of worst-case beliefs induce that sort of randomization.

For instance, what sort of uncertainty around a utility function makes an agent softmax over it? I don't know. Although, to be fair, I haven't tried particularly hard.

Generalization of Quantilization

However, one example that can be worked out (and has already been worked out by Jessica Taylor and Vanessa Kosoy) is quantilization.

If you haven't seen quantilization before, it's where you take a reference distribution over actions, $\nu \in \Delta X$, and instead of picking the best action to optimize the function $U$, you condition $\nu$ on the event "the action I picked is in the top 1 percent of actions sampled from $\nu$, in terms of how much $U$ likes it" and sample from that distribution.

I mean, it doesn't have to be 1 percent specifically, but this general process of "take a reference distribution, plot out how good it is w.r.t. some function $U$, condition on the $U$ score being beyond some threshold, and sample from that updated distribution" is referred to as quantilization.

Well, if that's the sort of optimization behavior we're looking for, we might ask "what sort of concave scoring function on probability distributions has quantilization as the optimum?", or "what sort of utility function uncertainty produces that scoring function?"

As it turns out, quantilization w.r.t. a reference distribution $\nu$, where your utility function is $U$, corresponds to worst-casing amongst the following set of functions for some $ϵ$. $$\{V|\int |U-V|d\nu \le ϵ\}$$ The epistemic state that leads to quantilization is therefore one where you think that your utility function is unfixably corrupted, but that its deviation from the true utility function is low relative to a given reference distribution $\nu$. Specifically, if $U$ is your utility function you see, and $V$ is the true utility function which $U$ is a corruption of, you believe that $\int |U-V|d\nu \le ϵ$ and have no other beliefs about the true utility function $V$.

You'd be very wary about going off-distribution because you'd think "the corruption can be arbitrarily bad off-distribution because all I believe is that $\int |U-V|d\nu$ is low, so in a region where $\nu$ is low-probability, it's possible that $V$ is super-low there". You'd also be very wary about deterministically picking a single spot where $U$ is high, because maybe the corruption is concentrated on those sparse few spots where $U$ is the highest.

However, if you randomize uniformly amongst the top quantile of $\nu$ where $U$ is high, this is actually the optimal response to this sort of utility function corruption. No matter how the corruption is set up (as long as it's small relative to $\nu$), the quantilizing policy is unlikely to do poorly w.r.t (the agent's irreducibly uncertain beliefs about) the true utility function.

These are some rather big claims. Exactly as stated above, they've already been proven a long time ago by people who aren't me. However, there's a further-reaching generalization that the above results are a special case of, which allows interpolating between quantilization and maximization, which is novel.

Introduction to Lp Spaces (skippable)

If you already know what $L_p$ spaces are, you can skip straight to the general theorem, but if you don't, time to take a detour and explain them.

Given some nice space $X$ and probability distribution $\nu \in \Delta X$, we can make the vector space of "measurable functions $X\to R$ which are equivalent w.r.t. $\nu$". After all, you can add functions together, and multiply by constants, so it makes a vector space. However, the elements of this vector space aren't quite functions, they're equivalence classes of functions that are equivalent w.r.t. $\nu$. Ie, if $f$ and $g$ are two different functions, but $\nu$ has zero probability of selecting a point $x$ where $f\left(x\right)\ne g\left(x\right)$, then $f$ and $g$ will be the same point in the vector space we're constructing.

This vector space can be equipped with a norm. Actually, it can be equipped with a lot of norms. One for each real number $p$ in $[1,\infty ]$. The $L_p$ norm on the space of "functions that are equivalent w.r.t. $\nu$" is: $$||f|{|}_p:={(\int |f{|}^{p}d\nu )}^{\frac{1}{p}}$$ For the $L_2$ norm, it'd be $$||f|{|}_2:=\sqrt{\int f^{2}d\nu }$$ Compare this to euclidean distance! $$||x||=\sqrt{\sum _{i=1}^n{x}_i^2}$$ So, our set-of-functions which induces quantilization behavior, $$\{V|\int |U-V|d\nu \le ϵ\}$$ Can also be expressed as "an $ϵ$-sized ball around $U$ w.r.t. the $L_1$ norm and $\nu$", and this is the core of how to generalize further, for we may ask, what's so special about the $L_1$ norm? What about the $L_p$ norm for all the $p\in [1,\infty ]$? What do those do?

Theorem 2: $L_p$ Ball Theorem

For any $ϵ>0$, $p\in [1,\infty ]$, $f:X\to R$, and $\nu :\Delta X$, the infrafunction corresponding to $\{g|{(\int |f-g{|}^{p}d\nu )}^{\frac{1}{p}}\le ϵ\}$ (Knightian uncertainty over the $L_p$ ball of size $ϵ$ centered at $f$, w.r.t $\nu$), is $\mu \mapsto \mu \left(f\right)-ϵ||\frac{d\mu }{d\nu }|{|}_q$ where $\frac{1}{p}+\frac{1}{q}=1$. Further, given a function $f$, the optimal $\mu$ to pick is the distribution $a\cdot \nu \cdot max(f-b,0{)}^{p-1}$, for some constants $a>0$ and $b\le max\left(f\right)$.

Some special cases of this are as follows. For $p=1$, you get quantilization. Worst-casing over little $L_1$ balls means that your task is to pick the probability distribution $\mu$ which maximizes $\mu \left(f\right)-ϵ||\frac{d\mu }{d\nu }|{|}_{\infty }$, and this maximizing probability distribution is $\nu$ rescaled by the function that's 1 when $f$ exceeds the threshold value and 0 otherwise (as this is the limit of $max(f-b,0{)}^{\delta }$ as $\delta \to 0$) This can be restated as conditioning $\nu$ on the event that $f$ exceeds a certain threshold, and so we get quantilizers.

For $p=2$, you get a more aggressive sort of optimization. Worst-casing over little $L_2$ balls means that your task is to pick the probability distribution $\mu$ which maximizes $\mu \left(f\right)-ϵ||\frac{d\mu }{d\nu }|{|}_2$, and this maximizing probability distribution is $\nu$ but rescaled linearly with how much $f$ exceeds the threshold value $b$. So, for example, given two points $x$ and $y$, if $f\left(x\right)-b=3\left(f\right(y)-b)$, the probability density at $x$ is enhanced by a factor of 3 over what it'd be at $y$.

For $p=\infty$, you basically just get argmax over the support of the distribution $\nu$. Worst-casing over little $L_{\infty }$ balls means that your task is to pick the probability distribution $\mu$ which maximizes $\mu \left(f\right)-ϵ||\frac{d\mu }{d\nu }|{|}_1$, and this maximizing probability distribution is $\nu$ but rescaled according to $max(f-b,0{)}^n$ for arbitrarily large $n$. Ie, incredibly large at the highest value of $f$, which dominates the other values. So pretty much, a dirac-delta distribution at the best point in the support of $\nu$.

There's a massive amount of work to be done on how various sorts of randomization behavior from agents relate to various sorts of concave scoring rules for distributions, and how those relate with various sorts of (restricted) worst-case assumptions about how the utility function got corrupted.

Dynamic Consistency??

But what's this about agents with infrafunctions $\text{U}$ as their utility functions being stable under reflection? Well, we'd want an agent to be incentivized to keep its utility function the same (or at least not change it in unpredictable ways) no matter what it sees. Making this more precise, if an agent has a utility (infra)function $\text{U}$, then it should believe that optimizing for the infrafunction $\text{U}|h$ ($\text{U}$ but modified in a predictable way to account for having seen history $h$) after seeing history $h$, will produce equal or better results (according to $\text{U}$) than optimizing for any competitor (infra)function $\text{V}$ after seeing $h$. This is a necessary condition to give the starting agent no incentive to alter the utility function of its future self in an unpredictable way (ie, alter it in a way that differs from $\text{U}|h$).

For example, if an agent with an infrafunction $\text{U}$ ever ends up thinking "If I was an optimizer for the utility function (not infrafunction!) $V$, I'd do better, hang on, lemme just rewrite myself to optimize that instead", that would be instability under reflection. That just should not ever happen. Infrafunctions shouldn't collapse into utility functions.

And, as it turns out, if you've got an agent operating an an environment with a utility infrafunction, there is a way to update the infrafunction over time which makes this happen. The agent won't want to change its infrafunction in any way other than by updating it. However, the update shares an undesirable property with the dynamically consistent way of updating infradistributions, though. Specifically, the way to update a utility infrafunction (after you've seen a history) depends on what the agent's policy would do in other branches.

If you're wondering why the heck we need to update our utility infrafunction over time, and why updating would require knowing what happens in alternate timelines, here's why. The agent is optimizing worst-case expected value of the functions within its set-of-functions. Thus, the agent will tend to focus its marginal efforts on optimizing for the utility functions in its set which have the lowest expected value, in ways that don't destroy too much value for the utility functions which are already doing well in expectation. And so, for a given function $f\in \hat{\text{U}}$ (the set induced by the infrafunction $\text{U}$), it matters very much whether the agent is doing quite well according to $f$ in alternate branches (it's doing well in expectation so it's safe to mostly ignore it in this branch), or whether the agent is scoring horribly according to $f$ in the alternate branches (which means that it needs to be optimized in this branch).

Time to introduce the notation to express how the update works. If $h$ is a finite history, and ${\pi }_{\neg h}$ is a stochastic partial policy that tells the agent what to do in all situations except where the history has $h$ as a prefix, and ${\pi }^{\ast }$ is a stochastic partial policy that tells the agent what to do in all situations where the history has $h$ as a prefix, then ${\pi }^{\ast }\bullet {\pi }_{\neg h}$ is the overall policy made by gluing together those two partial policies.

Also, if $e$ is an environment, and $\pi$ is a stochastic policy, $\pi \cdot e$ refers to the distribution over infinite histories produced by the policy interacting with the environment. By abuse of notation, ${\pi }_{\neg h}\cdot e$ can be interpreted as a probability distribution on $$\{h^{\prime }\in (A\times O{)}^{\omega }|h⋢h^{\prime }\}\cup \{h\}$$ This is because, obeying ${\pi }_{\neg h}$, everything either works just fine and ${\pi }_{\neg h}$ keeps telling you what your next action is and you build some infinite history $h^{\prime }$, or the partial history $h$ happens and ${\pi }_{\neg h}$ stops telling you what to do.

The notation ${\text{1}}_{\neg h}$ is the indicator function that's 1 when the full history lacks $h$ as a prefix, and 0 on the partial history $h$. $U_{\downarrow h}$ is the function $U$, but with a restricted domain so it's only defined on infinite bitstrings with $h$ as a prefix.

With those notations out of the way, given an infrafunction $\text{U}$ (and using $U$ for the utility functions in the corresponding set $\hat{\text{U}}$), we can finally say how to define the updated form of $\text{U}$, where we're updating on some arbitrary history $h$, environment $e$, and off-history partial stochastic policy ${\pi }_{\neg h}$ which tells us how we act for all histories that lack $h$ as a prefix.

Definition 1: Infrafunction Update For an infrafunction $\text{U}$ of type $\Delta (A\times O{)}^{\omega }\to R$, history $h$, environment $e$, and partial stochastic policy ${\pi }_{\neg h}$ which specifies all aspects of how the agent behaves except after history $h$, $\text{U}|e,h,{\pi }_{\neg h}$, the update of the infrafunction, is the infrafunction corresponding to the set of functions $$\left\{\right({\pi }_{\neg h}\cdot e\left)\right({\text{1}}_h)\cdot U_{\downarrow h}+({\pi }_{\neg h}\cdot e\left)\right({\text{1}}_{\neg h}U)|U\in \hat{\text{U}}\}$$ Or, restated, $$\left\{E_{h^{\prime }\sim {\pi }_{\neg h}\cdot e}\right[{\text{1}}_h]\cdot U_{\downarrow h}+E_{h^{\prime }\sim {\pi }_{\neg h}\cdot e}[{\text{1}}_{\neg h}U]|U\in \hat{\text{U}}\}$$

So, basically, what this is doing is it's taking all the component functions, restricting them to just be about what happens beyond the partial history $h$, scaling them down, and taking the behavior of those functions off-h to determine what constant to add to the new function. So, functions which do well off-h get a larger constant added to them than functions which do poorly off-h.

And now we get to our theorem, that if you update infrafunctions in this way, it's always better (from the perspective of the start of time) to optimize for the updated infrafunction than to go off and rewrite yourself to optimize for something else.

Theorem 3: Dynamic Consistency For any environment $e$, finite history $h$, off-history policy ${\pi }_{\neg h}$, and infrafunctions $\text{U}$ and $\text{V}$, we have that $$\text{U}(({\pi }_{\neg h}\bullet {\text{argmax}}_{{\pi }^{\ast }}(\text{U}|e,h,{\pi }_{\neg h}\left)\right({\pi }^{\ast }\cdot \left(e|h\right)\left)\right)\cdot e)\ge \text{U}(({\pi }_{\neg h}\bullet {\text{argmax}}_{{\pi }^{\ast }}\text{V}({\pi }^{\ast }\cdot \left(e|h\right)\left)\right)\cdot e)$$ Or, restating in words, selecting the after-h policy by argmaxing for $\text{U}|e,h,{\pi }_{\neg h}$ makes an overall policy that outscores the policy you get by selecting the after-h policy to argmax for $\text{V}$.

Ok, but what does this sort of update mean in practice? Well, intuitively, if you're optimizing according to an infrafunction, and some of the component functions you're worst-casing over are sufficiently well-satisfied in other branches, they kind of "drop out". We're optimizing the worst-case, so the functions that are doing pretty well elsewhere can be ignored as long as you're not acting disastrously with respect to them. You're willing to take a hit according to those functions, in order to do well according to the functions that aren't being well-satisfied in other branches.

Why Worst Case?

Worst case seems a bit sketchy. Aren't there more sane things to do like, have a probability distribution on utility functions, and combine them according to geometric average? That's what Nash Bargaining does to aggregate a bunch of utility functions into one! Scott Garrabrant wrote an entire sequence about that sort of stuff!

Well, guess what, it fits in the infrafunction framework. Geometric averaging of utility functions ends up being writeable as an infrafunction! (But I don't know what it corresponds to worst-casing over). First up, a handy little result.

Proposition 2: $L_p$ Double Integral Inequality If $f\ge 0$, let ${\oint }^{p}fd\mu$ be an abbreviation for ${(\int f^{p}d\mu )}^{\frac{1}{p}}$. Then for all $p\in [-\infty ,1]$, and $\mu :\Delta X$ and $\nu :\Delta Y$ and $f:X\times Y\to R^{\ge 0}$, we have that $$\int {\oint }^{p}f(x,y)d\nu d\mu \le {\oint }^p\int f(x,y)d\mu d\nu$$

Corollary 1: $L_p$-Averaging is Well-Defined Given any distribution $\nu$ over a family of functions or infrafunctions ${\text{F}}_i\ge 0$, define the $L_p$-average of this family (for $p\in [-\infty ,1]$) as the function $$\mu \mapsto {\oint }^p{\text{F}}_i\left(\mu \right)d\nu$$ $L_p$-averaging always produces an infrafunction.

Corollary 2: Geometric Mean Makes Infrafunctions The geometric mean of a distribution $\nu$ over utility functions is an infrafunction.

Proof: The geometric mean of a distribution over utility functions $U_i$ is the function $$\mu \mapsto e^{\int \ln \left(U_i\right(\mu \left)\right)d\nu }$$ However, the geometric mean is the same as the $L_0$ integral. So we get that it's actually writeable as $$\mu \mapsto {\oint }^0{U}_i\left(\mu \right)d\nu$$ And we can apply Corollary 2 to get that it's an infrafunction.

So, this $L_p$-mixing, for $p\in [-\infty ,1]$ is... well, for 1, it's just usual mixing. For 0, it's taking the geometric average of the functions. For $-\infty$, it's taking the minimum of all the functions. So, it provides a nice way to interpolate between minimization, geometric averaging, and arithmetic averaging, and all these ways of aggregating functions produce infrafunctions.

Just don't ask me what utility functions are actually in the infrafunction corresponding to a geometric mixture.

The Crappy Optimizer Theorem

Technically this theorem doesn't actually belong in this post. But it's close enough to the subject matter of this post to throw it in anyways. It turns out that "every" (not really) vaguely optimizer-ish process can be reexpressed as some sort of ultradistribution. An ultradistribution is basically an infradistribution (a closed convex set of probability distributions), except it maximizes functions instead of minimizing them.

And so, "every" (not really) optimizer-y process can be thought of as just argmax operating over a more restricted set of probability distributions.

Try not to read too much into the Crappy Optimizer Theorem, I'd very strongly advise that you take your favorite non-argmax process and work out how it violates the assumptions of the theorem. Hopefully that'll stop you from thinking this theorem is the final word on optimization processes.

Anyways, let's discuss this. The type signature of argmax is $(X\to R)\to X$. Let's say we're looking for some new sort of optimizer that isn't argmax. We want a function $s$ of the same type signature, that "doesn't try as hard".

We don't actually know what $s$ is! It could be anything. However, there's a function $Q:\left(\right(X\to R)\to X)\to \left(\right(X\to R)\to R)$, which I'll call the "score-shift" function, and it's defined as follows. $$s\mapsto (f\mapsto f(s\left(f\right)\left)\right)$$ Basically, given an optimizer and a function, you run the optimizer on the function to get a good input, and shove that input through the function to get a score. As a concrete example, $Q\left(\text{argmax}\right)=\text{max}$. If you have a function $f$, argmax over it, and plug the result of argmax back into $f$, that's the same as taking the function $f$ and producing a score of $max\left(f\right)$.

So, instead of studying the magical optimizer black box $s$, we'll be studying $Q\left(s\right)$ instead, and characterizing the optimization process by what scores it attains on various functions. There are four properties in particular, which it seems like any good optimizer should fulfill.

1: $c$-additivity For any constant function $c$ and function $f$, $Q\left(s\right)(f+c)=Q\left(s\right)\left(f\right)+c$.

2: Homogenity For any $a\ge 0$ and function $f$, $Q\left(s\right)(a\cdot f)=a\cdot Q\left(s\right)\left(f\right)$.

3: Subadditivity For any functions $f,g$, $Q\left(s\right)(f+g)\le Q\left(s\right)\left(f\right)+Q\left(s\right)\left(g\right)$.

4: Zero bound For any function $f\le 0$, $Q\left(s\right)\left(f\right)\le 0$.

Rephrasing this, though we don't know what the optimization-y process $s$ is, it's quite plausible that it'll fulfill the following four properties.

1: If you add a constant to the input function, you'll get that constant added to your score.

2: If you rescale the input function, it rescales the score the optimization process attains.

3: Optimizing the sum of two functions does worse than optimizing them separately and adding your best scores together.

4: Optimizing a function that's never positive can't produce a positive score.

Exercise: Which of these properties does softmax break? Which of these properties does gradient ascent with infinitesimal step-size break?

Also, notice that the first two properties combined are effectively saying "if you try to optimize a utility function, the optimization process will ignore scales and shifts in that utility function".

Theorem 4: Crappy Optimizer Theorem For any selection process $s$ where $Q\left(s\right)$ fulfills the four properties above, $\forall f:Q\left(s\right)\left(f\right)={max}_{\mu \in \Psi }\mu \left(f\right)$ will hold for some closed convex set of probability distributions $\Psi$. Conversely, the function $f\mapsto {max}_{\mu \in \Psi }\mu \left(f\right)$ for any closed convex set $\Psi$ will fulfill the four properties of an optimization process.

Informal Corollary 3: Any selection process $s$ where $Q\left(s\right)$ fulfills the four properties is effectively just argmax but using a restricted set of probability distributions.

Other Aspects of Infrafunctions

Infrafunctions are the analogue of random variables in inframeasure theory. Here are two useful properties of them.

First off, in the classical case of functions, we can average functions together. There's a distinguished averaging function $\Delta (X\to R)\to (X\to R)$. Pretty obvious.

When we go to infrafunctions, this gets extended somewhat. There's a distinguished function $\square FX\to FX$, where $\square$ is the space of infradistributions, and $FX$ is the space of infrafunctions. If you know enough category theory, we can say that the space of infrafunctions is a $\square$-algebra, where $\square$ is the infradistribution monad. If you don't know enough category theory, basically, there's an infra-version of "averaging points together" and it makes all the diagrams commute really nicely.

Proposition 3: The space of infrafunctions is a $\square$-algebra, with the function $fla{t}_{FX}:\square FX\to FX$ being defined as $\lambda \psi .\lambda \mu .\psi (\lambda \text{F}.\text{F}(\mu \left)\right)$.

Also, in the classical case, if you've got a function $g:X\to Y$, that produces a function $(Y\to R)\to (X\to R)$ in a pretty obvious way. Just precompose. $f\mapsto (x\mapsto f(g\left(x\right)\left)\right)$.

A similar thing holds here. Except, in this case, instead of a function, we can generalize further to a continuous infrakernel $k:X\to \square Y$. Again, to get a function $FY\to FX$, you just precompose. $\text{G}\mapsto (\mu \mapsto \text{G}(k_{\ast }\left(\mu \right))$. Take the distribution $\mu$, shove it through the infrakernel $k$ to get an infradistribution on $Y$, and shove that through the infrafunction $\text{G}$.

Proposition 4: All continuous infrakernels $k:X\to \square Y$ induce a function $FY\to FX$ via $\text{G}\mapsto (\mu \mapsto \text{G}(k_{\ast }\left(\mu \right))$

So, given an infrakernel (and functions are a special case of this) going one way, you can transfer infrafunctions backwards from one space to the other.

The restriction to continuous infrakernels, though, actually does matter here. A function $f:X\to Y$ induces two infrakernels, one of type $X\to \square Y$ (the image), and one of type $Y\to \square X$ (the preimage). So, theoretically we could get a function of type $(X\to Y)\to (FX\to FY)$ by routing through the preimage function. However, since it requires continuity of the function $Y\to \square X$ to make things work out, you can only reverse the direction if the function mapping a point $x$ to its preimage is Hausdorff-continuous. So, for functions with Hausdorff-continuous inverses, you can flip the usual direction and go $(X\to Y)\to (FX\to FY)$. But this trick doesn't work in general, only $(X\to \square Y)\to (FY\to FX)$ is valid in general.

There's a bunch of other things you can do, like intersection of infrafunctions making a new infrafunction, and union making a new infrafunction. Really, most of the same stuff as works with infradistributions.

The field is wide-open. But it's a single framework that can accomodate Scott's generalized epistemic states, Scott's geometric averaging, Knightian uncertainty, intersecting and unioning that uncertainty, averaging, quantilizers, dynamic consistency, worst-case reasoning, and approximate maximization. So it seems quite promising for future use.