TL;DR: I verify that the minimal map lemma can be used as a criterion for functional optimization. Specifically, I derive a broad class of consistent loss functions according to the criterion. I also give an application of the result to determining how to train a neural representation.

Say we receive data $(x,y)\sim X\times \{-1,1\}$ where the $x$ are observations and $y$ are binary class labels. We wish to learn a map $h\in H$ that is able to properly classify the data. By the minimal map lemma this is information equivalent to learning the map $\zeta \circ p\left(y|x\right)$ where $\zeta$ is an arbitrary isomorphism. We can make a stylistic choice on the the model class by imposing the constraint, $${\zeta }^{-1}\left(h\right)+{\zeta }^{-1}(-h)=1$$ Consider the term $y\cdot h\left(x\right)$. If the prediction is correct then $y\cdot h\left(x\right)>0$ or otherwise $y\cdot h\left(x\right)<0$. However, we're either right or wrong with probability one. Therefore, if ${\zeta }^{-1}(y\cdot h(x\left)\right)>1/2$ that label and the classifier most likely agree. In other words, we should choose the $y$ that maximizes the probability of agreement.

In practice, this means we calculate, $$\iota (y\cdot h(x\left)\right)=\frac{1}{2}(1-\text{sign}(y\cdot h\left(x\right)\left)\right)$$ to collapse probabilities into binary decisions. The variable is threshold that returns one if the classification is less likely than the other and zero otherwise. This is effectively the zero-one loss.

Theorem: Consider the population risk, $$R_{\text{0-1}}\left(h\right)={\int }_{X\times Y}\iota \left(h\right)p\left(1|x\right)+\iota (-h)(1-p(1|x\left)\right)d\mu \left(x\right)$$ The optimal decision rule is given by $\text{sign}\left(h\right)=\text{sign}\left(p\right(1|x)-1/2)$.

Proof: We should set $\text{sign}\left(h\right)=1$ if $p\left(1|x\right)>1-p\left(1|x\right)$ which implies that $p\left(1|x\right)>1/2$. The result is that the risk is minimized. Thus, the optimal decision rule is given as stated.

As stated before, if ${\zeta }^{-1}(y\cdot h(x\left)\right)>1/2$ then we should take that as our classification. The trick now is to replace $\iota \to \varphi$ with the most general class of loss functions such that the optimal classifier remains unchanged. So we'll consider, $$R_{\varphi }\left(h\right)={\int }_{X\times Y}\varphi \circ {\zeta }^{-1}\left(h\right)\eta \left(x\right)+\varphi \circ {\zeta }^{-1}(-h)(1-\eta (x\left)\right)d\mu \left(x\right)$$ where we've introduced a twice differentiable loss function $\varphi :R\to R$ and collapsed our conditional probability to just the function $\eta$.

Definition: A loss function $\varphi$ is consistent if the optimal classifier for $R_{\varphi }$ is equivalent to the optimal classifier for $R_{\text{0-1}}$.

We'll also need a lemma to proceed,

Lemma: If the functional derivative ${\nabla }_{h}R$ exists, then $h^{\ast }$ is locally optimal iff, $$⟨{\nabla }_{h}R,h^{\ast }-h⟩\ge 0,\forall h\in H$$ Proof: This is the first order condition for optimality.

Now we can state the main result.

Theorem: A loss function $\varphi :R\to R$ is consistent if it is of the form, $$\varphi \left(\eta \right)=C\left(\eta \right)+(1-\eta )C^{\prime }\left(\eta \right)$$ where $C\left(\eta \right)=C(1-\eta )$ is a twice differentiable strictly convex function and $\eta ={\zeta }^{-1}\left(h\right(x\left)\right)$ respects $h$ as a minimal map according to the constraint, $${\zeta }^{-1}\left(h\right)+{\zeta }^{-1}(-h)=1$$ and for the optimal classifier $h^{\ast }$ satisfies, $${\zeta }^{-1}\circ h^{\ast }\left(x\right)=p\left(1|x\right)$$ Proof:

To minimize the population risk we need to use the functional deriviative. $${\nabla }_{h}R=\underset{ϵ\to 0}{lim}\frac{R(h+ϵ\psi )-R\left(h\right)}{ϵ}={\left[\frac{d}{dϵ}R\right(h+ϵ\psi \left)\right]}_{ϵ=0}={\int }_{X\times Y}[{\partial }_h\varphi \circ {\zeta }^{-1}(h\left)\eta \right(x)+{\partial }_h\varphi \circ {\zeta }^{-1}(-h\left)\right(1-\eta \left(x\right)\left)\right]\psi \left(x\right)d\mu \left(x\right)$$ Now, we want to fix a class of $\varphi$ with the assumption that $\text{sign}\left(h\right)$ will determine our classifier. Any particulars beyond this are irrelevant to producing a 0-1 optimal classifier. This simplifies the optimality condition to, $$\text{sign}({\partial }_h\varphi \circ {\zeta }^{-1}(h)\eta +{\partial }_h\varphi \circ {\zeta }^{-1}(-h\left)\right(1-\eta \left)\right)=\text{sign}\left(h\right)⟺\text{sign}\left({\varphi }^{\prime }\right(\hat{\eta }\left){\partial }_h{\zeta }^{-1}\right(h)\eta -{\varphi }^{\prime }(1-\hat{\eta }\left){\partial }_h{\zeta }^{-1}\right(h\left)\right(1-\eta \left)\right)=\text{sign}\left(h\right)⟺\text{sign}\left({\varphi }^{\prime }\right(\hat{\eta })\eta -{\varphi }^{\prime }(1-\hat{\eta }\left)\right(1-\eta \left)\right)=\text{sign}\left(h\right)$$ Note that we assume here that ${lim}_{h\to \infty }{\zeta }^{-1}\left(h\right)=1$ and is differentiable. If we have, $${\varphi }^{\prime }\left(\hat{\eta }\right)=f\left(\hat{\eta }\right)C^{\prime \prime }\left(\hat{\eta }\right)\text{s.t}{C}^{\prime \prime }\left(\eta \right)>0$$ then we can do this trick again. So our loss function is a function times a strictly convex function. Thus, $$\text{sign}\left({\varphi }^{\prime }\right(\hat{\eta })\eta -{\varphi }^{\prime }(1-\hat{\eta }\left)\right(1-\eta \left)\right)=\text{sign}\left(h\right)⟺\text{sign}\left(f\right(\hat{\eta }\left)C^{\prime \prime }\right(\hat{\eta })\eta -f(1-\hat{\eta }\left)C^{\prime \prime }\right(1-\hat{\eta }\left)\right(1-\eta \left)\right)=\text{sign}\left(h\right)⟺\text{sign}\left(f\right(\hat{\eta })\eta -f(1-\hat{\eta }\left)\right(1-\eta \left)\right)=\text{sign}\left(h\right)$$ Recall that $\hat{\eta }\left(x\right)={\zeta }^{-1}\left(h\right(x\left)\right)$ so at $\hat{\eta }=\eta$ we must be at an optimum if we're to respect the minimal map property and treat ${\zeta }^{-1}\circ h$ as a probability. This is the invertible link property. To satisfy this property we need, $$\frac{f\left(\hat{\eta }\right)}{f(1-\hat{\eta })}=\frac{1-\eta }{\eta }\Rightarrow f\left(\eta \right)\cong (1-\eta )\text{or}1/\eta$$ subsitution makes it clear that the solution we want is $f\left(\eta \right)=(1-\eta )$. The other solution is degenerate. We're almost done, now we solve for $\varphi$ with an integration by parts, $$\varphi \left(\eta \right)=\int (1-\eta )\cdot C^{\prime \prime }\left(\eta \right)=(1-\eta )C^{\prime }\left(\eta \right)+\int C^{\prime }\left(\eta \right)=C\left(\eta \right)+(1-\eta )C^{\prime }\left(\eta \right)$$ $\square$

Examples:

Take $$C\left(\eta \right)=H\left(\eta \right)$$ and $${\zeta }^{-1}\left(v\right)=\frac{1}{\log \left(2\right)}\frac{e^v}{1+e^v}$$ then we have,

$$\varphi \left(v\right)=\frac{1}{\log \left(2\right)}\log (1+e^v)$$

which is the logistic loss.

More abstractly, for a neural representation we compute layers as, $${\overrightarrow{h}}_{j+1}=\sigma \left(W_j{\overrightarrow{h}}_j\right)$$ where $\sigma$ is an activation for each layer ${\overrightarrow{h}}_j$. Previously I showed that if a neural network is to be interpreted as a probabalistic decision process then we need $\sigma =\zeta \circ p$ where $\zeta$ is invertible and $p$ can be an associated probability.

When we arrive at the output layer, we need to apply a loss function. Given the above analysis it would be best to change the activation to ${\zeta }^{-1}$ and then apply a consistent loss $\varphi$ from above. This ensures that what is going into $\varphi$ is indeed a probability.