In machine learning, one often wants to uniformly approximate an arbitrary continuous function arbitrarily well using polynomials, neural networks, or something else. But in the case of complex-valued functions, this is more difficult. For example, the limit of holomorphic functions in the topology of uniform convergence on compact sets is always holomorphic, so holomorphic functions cannot directly approximate non-holomorphic functions. But in this post, I will show you how one can approximate arbitrary continuous functions indirectly from something like a holomorphic function.

I will state and prove the result in the full generality which means I will need to use uniform algebras over compact sets, but I do not need to use the theory of uniform algebras in order to state and prove the result. I will try to make everything here self-contained.

I came up with the statement and the proof of the result myself.

Motivation: 

I am not too much of a fan of neural networks. Even though neural networks perform well in practice, they do not behave in a way that is too appealing to pure mathematicians. For example, if you try to feed purely imaginary inputs into a vanilla neural network with tanh activation and no bias, you will just obtain a random vector from some multi-variate Cauchy distribution as output.

Polynomials, on the other hand, behave in a way appealing to mathematicians, so it would be nice if multivariate polynomials had a stronger presence in machine learning. I have personally trained polynomial machine learning models and they seem to behave mathematically in the sense that if we train the model multiple times with different initializations, we tend to end up with the same trained model; these polynomial models that I have trained are somewhat sophisticated too since they can have multiple layers, but I have not developed them as well as neural networks.

To demonstrate how polynomials may be useful in machine learning, one may want to resort to a uniform approximation for polynomials, but the Stone-Weierstrass approximation theorem allows us to generalize such a uniform approximation theorem from polynomials to nearly arbitrary rings of real-valued functions on some compact Hausdorff space. In this post, we shall give another generalized uniform approximation theorem that applies to uniform algebras which are closed algebras of continuous functions on compact Hausdorff spaces.

In this post, we shall use quantum states to overcome the limitations of some classes of complex-valued functions in approximating arbitrary continuous functions. This use of quantum states alludes to a possible way that one can use quantum states and partial traces to improve the performance of machine learning models.

Uniform algebras:

Suppose that $X$ is a compact Hausdorff space. Let $C\left(X\right)$ denote the collection of all continuous functions $f:X\to C$. Give $C\left(X\right)$ the norm $\parallel \ast \parallel$ defined by$\parallel f\parallel =max\left\{|f\right(x)|:x\in X\}$. Then $C\left(X\right)$ is a Banach algebra. A closed subalgebra $A$ of $C\left(X\right)$ is said to be a uniform algebra if $A$ contains all constant functions and whenever $x,y\in X,x\ne y$, there is some $f\in A$ with $f\left(x\right)\ne f\left(y\right)$. 

Example 0: If $X$ is a compact Hausdorff space, then $C\left(X\right)$ is always a uniform algebra.

Example 1: If $K$ is a compact subset of $C^n$, then let $A$ be the set of all continuous functions $f:K\to C$ which are holomorphic on the interior of $A$. Then $A$ is a uniform algebra.

Example 2: Suppose that $B$ is a commutative Banach algebra. Let $X$ denote the set of all continuous Banach algebra homomorphisms $\varphi :B\to C$. Then $X$ becomes a compact Hausdorff space in the weak*-topology. If $a\in B$, then define a continuous function $\hat{a}:X\to C$ by setting $\hat{a}\left(\varphi \right)=\varphi \left(a\right)$. Then $\{\hat{a}:a\in B\}$ is a uniform algebra.

If $X$ is a compact Hausdorff space, $A\subseteq C\left(X\right)$ is a subspace algebra, and $V$ is a finite dimensional complex vector space. Then let $A\otimes V$ denote the set of all functions $f:X\to V$ where if $L:V\to C$  is linear , then $L\circ f\in A$.

Lemma: Suppose that $X$ is a compact Hausdorff space $A\subseteq C\left(X\right)$ is a linear subspace that contains all constant functions and where if $x,y\in X,x\ne y$, there is some $f\in A$ with $f\left(x\right)\ne f\left(y\right)$. Then whenever $x_0\in U\subseteq X$ and $U$ is open, there is some finite dimensional complex inner product space $V$ and $f\in A\otimes V$ with $f\left(x_0\right)=0$ and $\parallel f\left(y\right)\parallel >1$ whenever $y\in X\setminus U$.

Proof: The proof will just use a standard compactness argument. For each $y\in X\setminus U$, let $f\in A$ be a function with $|f_y\left(y\right)|>1$ and $f_y\left(x_0\right)=0$. Let $U_y=\{x\in X:|f_y(x)|>1\}$ for each $y\in X\setminus U$. Then by compactness, there are $y_1,\dots ,y_n\in X\setminus U$ where $X=U\cup U_{y_1}\cup \cdots \cup U_{y_n}$. Define $f:X\to C^n$ by setting $f\left(x\right)=\left(f_{y_1}\right(x),\dots ,f_{y_n}(x\left)\right)$. Then$f\left(x_0\right)=0$ and if $y\in X\setminus U$, there is some $k$ with $y\in U_{y_k}$, so in this case, $|f_{y_k}\left(y\right)|>1$, so $|f\left(y\right)|>1$ as well. Q.E.D.

Recall that a density operator is a positive semidefinite trace 1 linear operator. Given a finite dimensional complex inner product space $V$, let $D\left(V\right)$ denote the collection of all density operators $A:V\to V$, and let $L\left(V\right)$ denote the collection of all linear operators from $V$ to $V$. Suppose that $V,W$ are finite dimensional complex inner product spaces. Then the partial trace is the unique linear mapping ${\text{Tr}}_W:L(V\otimes W)\to L\left(V\right)$ subject to the condition that ${\text{Tr}}_W(R\otimes S)=R\cdot \text{Tr}\left(S\right)$ whenever $R:V\to V,S:W\to W$ are linear.

Theorem: Suppose that $X$ is a compact Hausdorff space, $A\subseteq C\left(X\right)$ is uniform algebra, $V$ is a finite dimensional complex inner product space and $f:X\to D\left(V\right)$ is a continuous function. Then whenever $ϵ>0$ and $\parallel \ast \parallel$ is a matrix norm, there is some finite dimensional complex inner product space $W$ and $g\in A\otimes V\otimes W$ where if $x\in X$,

then $\parallel f\left(x\right)-\frac{{\text{Tr}}_W\left(g\right(x)\cdot g(x{)}^{\ast })}{\parallel g\left(x\right){\parallel }_2^2}\parallel <ϵ$.

Proof: Suppose that $U$ is an open cover of $X$. For each $y\in X$, there is some $V_y\in U$, natural number $d\left(y\right)$, and some $f_y\in A\otimes C^{d\left(y\right)}$ with $f_y\left(y\right)=0$ and $\parallel f_y\left(z\right){\parallel }_{\infty }>2$whenever $z\in X\setminus V_y$. Therefore, let $U_y=\{x\in X:\parallel f_y(x){\parallel }_{\infty }<1/2\}$. Then by compactness, there are $y_1,\dots ,y_n$ where $X=U_{y_1}\cup \cdots \cup U_{y_n}$. Let $U_j=U_{y_j},V_j=V_{y_j}$.

Define $f_k=f_{y_k}$. Let $e$ be a unit vector orthogonal to each vector in complex Euclidean space.

Let $Z_k=C^{d\left(y_k\right)}+⟨e⟩$ for each $k$, and let $Z=Z_1\otimes \cdots \otimes Z_n$. Let $W_0$ be a finite dimensional complex inner product space, and let $t:\{y_1,\dots ,y_n\}\to V\otimes W_0$ be a function such that ${\text{Tr}}_{W_0}\left(t\right(y_k\left)t\right(y_k{)}^{\ast })=f(y_k)$ for all $k$. Let $N$ be a positive integer. Define a function $h:X\to Z$ by setting $h\left(x\right)=(e+f_1(x{)}^N)\otimes \cdots \otimes (e+f_n\left(x{)}^N\right)$ where the power is taken elementwise.

If $S$ is a finite set of natural numbers, then let $S\left[k\right]$ denote the $k$-th element of the set $S$. Set $\left[n\right]=\{1,\dots ,n\}$ for each natural number $n$. For each $S\subseteq \left[n\right]$, let $v_{j,S}$ be a unit vector where if $(j,S)\ne (k,T)$, then $⟨v_{j,S},v_{k,T}⟩=0$.

We shall now construct a linear function $I$ that maps $Z$ to some other vector space.

Suppose that $S\subseteq \left[n\right]$ with $|S|=m$ and $x_j\in C^{d\left(j\right)}$ for $j\in S$ and $x_j=e$ otherwise. Then set $I(x_1\otimes \cdots \otimes x_n)={\sum }_{j\in \left[n\right]\setminus S}t\left(y_j\right)\otimes x_{S\left[1\right]}\otimes \cdots \otimes x_{S\left[m\right]}\otimes v_{j,S}$. Define $g=I\circ h$.

Observe that $g\left(x\right)={\sum }_{S\subseteq \left[n\right]}{\sum }_{j\in \left[n\right]\setminus S}t\left(y_j\right)\otimes f_{S\left[1\right]}(x{)}^N\otimes \cdots \otimes f_{S\left[|S|\right]}(x{)}^N\otimes v_{j,S}$. We include the vector $v_{j,S}$ to make sure that the summands are all orthogonal.

Let $\text{PTr}$ denote the partial trace where we trace out all factors in the tensor product except for $V$. Then

$\text{PTr}\left(g\right(x\left)g\right(x{)}^{\ast })={\sum }_{S\subseteq \left[N\right]}{\sum }_{j\in \left[n\right]\setminus S}f(y_j)\cdot \parallel f_{S\left[1\right]}(x{)}^N{\parallel }_2^2\dots \parallel f_{S\left[|S|\right]}(x{)}^N{\parallel }_2^2$.

Suppose now that $ϵ>0$. Then suppose that whenever $U\in U$, if $x,y\in U$, then$\parallel f\left(x\right)-f\left(y\right)\parallel <ϵ$.

If $x\in U_k$ and $\parallel f\left(x\right)-f\left(y\right)\parallel \ge ϵ$, then $y\notin V_k$, so $\parallel f_k\left(y\right){\parallel }_{\infty }>2$.

When $N$ is large, the dominant terms in the sum for $\text{PTr}\left(f\right(x\left)f\right(x{)}^{\ast })$ are indexed by sets $S$ where $k\notin S$ but where $S$ contains all elements $j$ with $\parallel f\left(x\right)-f\left(y_j\right)\parallel \ge ϵ$. Therefore, the value $\text{PTr}\left(g\right(x\left)g\right(x{)}^{\ast })/\parallel g(x){\parallel }_2^2$ can be uniformly approximated by $f\left(x\right)$.

Q.E.D.

Conclusion

The proof of the above theorem consists of a standard compactness argument that any mathematician who specializes in analysis should be able to come up with. The above result also applies to the field of real numbers since I nowhere mentioned anything about complex numbers, but for the real numbers, the above result is a consequence of the Stone-Weierstrass theorem. The Stone-Weierstrass theorem also applies to quaternionic-valued functions, so it is unnecessary to state the result for quaternions (and the proof will need to be restated and rewritten since quaternionic tensor products don't work). One should therefore not be too surprised by the above theorem. 

The above theorem may also be a corollary of a known result on Banach algebras (or something like that), but I was not able to find an appropriate reference.