There have been some arguments coming from MIRI that we should be designing AIs that are good at e.g. engineering while not knowing much about humans, so that the AI cannot manipulate or deceive us. Here is an attempt at a formal model of the problem.

We want algorithms that learn domain D while gaining as little as possible knowledge about domain E. For simplicity, let's assume the offline learning setting. Domain D is represented by instance space $X$, label space $Y$, distribution $\mu \in \Delta (X\times Y)$ and loss function $L:Y\times Y\to R$. Similarly, domain E is represented by instance space $Z$, label space $W$, distribution $\nu \in \Delta (Z\times W)$ and loss function $M:W\times W\to R$. The distributions $\mu ,\nu$ are initially unknown and we assume some prior over them: $\zeta \in \Delta \left(\Delta \right(X\times Y)\times \Delta (Z\times W\left)\right)$. The prior involves some correlation between D and E, hence learning about D tends to acquire information about E as well.

A learning algorithm $A$ for D is $A:(X\times Y{)}^{\ast }\to Y^X$ (receives a data sample and produces a label prediction function). A learning algorithm $B$ for E has access to knowledge generated by $A$: $B:Y^X\times (Z\times W{)}^{\ast }\times Z\to W$. We can now consider zero-sum games of the following form: you choose an algorithm $A$, the adversary looks at $A$ and chooses an algorithm $B$, your payoff decreases with your expected loss $E{L}^A$ and increases with the adversary's expected loss $E{M}^B$ (e.g. it is given by $-E{L}^A+\alpha E{M}^B$ for some parameter $\alpha >0$). The expected losses are given by

$$E{L}_n^A:=E_{(\mu ,\nu )\sim \zeta ,S\sim {\mu }^n,(x,y)\sim \mu }\left[L\right(y,A\left(S\right)\left(x\right)\left)\right]$$

$$E{M}_m^B:=E_{(\mu ,\nu )\sim \zeta ,S\sim {\mu }^n,T\sim {\nu }^m,(z,w)\sim \nu }\left[M\right(z,B\left(A\right(S),T,z)\left)\right]$$

Here $n,m\in N$ are the sample sizes. The interesting case is $n\gg m$ or even $m=0$.

Here's a very simple example. Suppose that $Y=[0,1]$, $L(y,y^{\prime })=(y-y^{\prime }{)}^2$ and $\zeta$ is s.t. (i) each $x\in X$ is assigned a persistent label sampled uniformly at random from $\{0,1\}$ independently of other instances (ii) information about the labels doesn't help with E but information about the distribution on $X$ does help with E. When we care only about $E{L}^A$ the best we can do is memorize the samples, i.e. set $A\left(S\right)\left(x\right)$ to $y$ if $(x,y)\in S$ and set it to $\frac{1}{2}$ otherwise. However, this would aid the adversary. Instead, we can set $A\left(S\right)\left(x\right)$ to $y$ if $(x,y)\in S$ and set it to a coinflip otherwise. Now our loss is somewhat worse (but, for discrete $X$ it still goes to $0$ as $n$ goes to $\infty$) but the adversary gains no information from us!

It is also possible to ignore any knowledge we have about E and just try designing $A$ which simultaneously minimizes the mutual information between $S$ and $A\left(S\right)$ and minimizes $E{L}^A$. Going to an even higher level of abstraction, this is similar to the following problem:

Let $(L,R,E)$ be a bipartite graph ($L$ are the left vertices, $R$ are the right vertices, $E\subseteq L\times R$ are the edges) and $\zeta$ a distribution on $L$. Find $f:L\to R$ s.t. (i) for any $v\in L$, $(v,f(v\left)\right)\in E$ and (ii) if we sample $v$ from $\zeta$ then the mutual information between $v$ and $f\left(v\right)$ is minimal. That is, we are minimizing the following:

$$I\left(f\right):=E_{v\sim \zeta }\left[\ln \frac{1}{\zeta \left(f^{-1}\right(f\left(v\right)\left)\right)}\right]$$

It would be interesting to understand the computational complexity of this problem (and/or of relaxations when we're allowed to approximate).

Finally, it is interesting to also impose computational complexity constraints on our $A$ (but perhaps not on $B$: obfuscating the learned representation means the knowledge about E is inaccessible from outside but might be still exploitable by the AI itself), in which case we would split it into a representation space $R_A$, a training algorithm algorithm $t_A:(X\times Y{)}^{\ast }\to R$ and a prediction algorithm $p_A:R\times X\to Y$ (both of which have to lie in some low complexity class e.g. $P$), whereas the signature of $B$ becomes $B:R\times (Z\times W{)}^{\ast }\times Z\to W$.