Joseph Van Name — LessWrong

LESSWRONG
LW

Approximating arbitrary complex-valued continuous functions

3mo

In machine learning, one often wants to uniformly approximate an arbitrary continuous function arbitrarily well using polynomials, neural networks, or something else. But in the case of complex-valued functions, this is more difficult. For example, the limit of holomorphic functions in the topology of uniform convergence on compact sets is always holomorphic, so holomorphic functions cannot directly approximate non-holomorphic functions. But in this post, I will show you how one can approximate arbitrary continuous functions indirectly from something like a holomorphic function.

I will state and prove the result in the full generality which means I will need to use uniform algebras over compact sets, but I do not need to use the... (read 1214 more words →)

Using complex polynomials to approximate arbitrary continuous functions

Joseph Van Name

4mo

Machine learning algorithms such as neural networks are supposed to have some sort of universal uniform approximation theorem that shows that they can (at least in principle) learn any possible data set without simply overfitting to the training data.

The standard universal approximation theorem applies to shallow neural networks with arbitrary continuous non-polynomial activation functions. There are also plenty of polynomial approximation results in mathematics, so one should at least in principle be able to train a real multivariate polynomial to model an arbitrary continuous function arbitrarily well using a multi-layered machine learning model. On the other hand, anyone modestly familiar with complex analysis knows that not every continuous function from a compact... (read 1401 more words →)

Spectral radii dimensionality reduction computed without gradient calculations

Joseph Van Name

9mo

In this post, I shall describe a fitness function that can be locally maximized without gradient computations. This fitness function is my own. I initially developed this fitness function in order to evaluate block ciphers for cryptocurrency technologies, but I later found that this fitness function may be used to solve other problems such as the clique problem (which is NP-complete) in the average case and some natural language processing tasks as well. After describing algorithms for locally maximizing this fitness function, I conclude that such a fitness function is inherently interpretable and mathematical which is what we need for AI safety.

Let $K$ denote either the field of real numbers, complex numbers, or the... (read 1590 more words →)

Joseph Van Name's Shortform

Joseph Van Name

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Interpreting a matrix-valued word embedding with a mathematically proven characterization of all optima

Joseph Van Name

In this post, I shall first describe a new word embedding algorithm that I came up with called a matrix product optimized (MPO) word embedding, and I will prove a theorem that completely interprets this word embedding in the simplest case. While it is probably infeasible to mathematically characterize a word embedding with a mathematical proof when the corpus is something that one encounters in practice, this theorem should be a signal that such a word embedding (or a similar word embedding) should be interpretable and mathematical in other ways as well. This theorem also illustrates the way that MPO word embeddings should behave.

Unlike most word embedding algorithms, MPO word embeddings are... (read 3412 more words →)

Interpreting a dimensionality reduction of a collection of matrices as two positive semidefinite block diagonal matrices

Joseph Van Name

Here are some empirical observations that I have made on August 14, 2023 to August 19, 2023 that are characteristics of the interpretability of my own matrix dimensionality reduction algorithm. These phenomena that we observe do not occur on all inputs (they sometimes occur partially); and it would be nice if there were a more complete mathematical theory with proofs that explains these empirical phenomena.

Given a (possibly directed and possibly weighed) graph with $n$ nodes represented as a collection of $n \times n$ -matrices $A_{1}, \dots, A_{r}$ , we will observe that a dimensionality reduction $(X_{1}, \dots, X_{r})$ of $(A_{1}, \dots, A_{r})$ where each $X_{i}$ is a $d \times d$ -matrix (I call this dimensionality reduction an LSRDR) is in many cases the optimal solution to a combinatorial problem for the graph. In this case, we... (read 1426 more words →)

When performing a dimensionality reduction on tensors, the trace is often zero.

Joseph Van Name

In this post, we shall define my new dimensionality reduction for tensors in $V^{\otimes n}$ where $n \geq 3$ , and we shall make an empirical observation about the structure of the dimensionality reduction. There are various simple ways of adapting this dimensionality reduction algorithm to tensors in $V_{1} \otimes \dots \otimes V_{n}$ and even mixed quantum states (mixed states are just positive semidefinite matrices in $V_{1} \otimes \dots \otimes V_{n}$ which trace 1), but that will be a topic for another post.

This dimensionality reduction shall represent tensors in $V^{\otimes n}$ as tuples of matrices $A_{1}, \dots, A_{r}$ . Computer experiments indicate that, in many cases, we have $Tr (A_{i_{1}} \dots A_{i_{m}}) = 0$ whenever $m \neq 0 mod n$ .

If $X$ is a matrix, then the spectral radius of $X$ is the value

$ρ (X) = max {| λ | : λ is an eigenvalue of X} = {lim}_{n \to \infty} ∥ A^{n} ∥^{1 / n}$ .

If $X$ is a matrix, then define the conjugate matrix $¯ ¯¯¯ ¯ X = (X^{*})^{T} = (X^{T})^{*}$ ; this is the matrix obtained from $X$ by replacing each entry with its... (read 773 more words →)