The intersection of machine learning and generalizations of Laver tables seems to have a lot of potential, so your question about this is extremely interesting to me.

Machine learning models from generalized Laver tables?

I have not been able to find any machine learning models that are based on generalized Laver tables that are used for solving problems unrelated to Laver tables. I currently do not directly see any new kinds of deep learning models that one can produce from generalizations of Laver tables, but I would not be surprised if there were a non-standard machine learning model from Laver like algebras that we could use for something like NLP. 

But I have to be skeptical about applying machine learning to generalized Laver tables. Generalizations of Laver tables have intricate structure and complexity but this complexity is ordered. Furthermore, there is a very large supply of Laver-like algebras (even with 2 generators) for potentially any situation. Laver-like algebras are also very easy to compute. While backtracking may potentially take an exponential amount of time, when I have computed Laver-like algebras, the backtracking algorithm seems to always terminate quickly unless it is producing a large quantity or Laver-like algebras or larger Laver-like algebras. But with all this being said, Laver-like algebras seem to lack the steerability that we need to apply these algebras to solving real-world problems. For example, try interpreting (in a model theoretic sense) modular arithmetic modulo 13 in a Laver-like algebra. That is quite hard to do because Laver-like algebras are not built for that sort of thing.

Here are a couple of avenues that I see as most promising for producing new machine learning algorithms from Laver-like algebras and other structures.

Let $X$ be a set, and let $\ast$ be a binary operation on $X$ that satisfies the self-distributivity identity $x\ast (y\ast z)=(x\ast y)\ast (x\ast z)$. Define the right powers $x^{\left[n\right]}$ for $n\ge 1$ inductively by setting $x^{\left[1\right]}=x$ and $x^{[n+1]}=x\ast x^{\left[n\right]}$. We say that $(X,\ast ,1)$ is a reduced nilpotent self-distributive algebra if $1$ satisfies the identities $1\ast x=x,x\ast 1=1$ for $x\in X$ and if for all $x\in X$ there is an $n\ge 1$ with $x^{\left[n\right]}=1$. A reduced Laver-like algebra is a reduced nilpotent self-distributive algebra $(X,\ast )$ where if $x_n\in X$ for each $n\ge 0$, then $x_0\ast \cdots \ast x_N=1$ for some $N$. Here we make the convention to put the implied parentheses on the left so that $a\ast b\ast c\ast d=\left(\right(a\ast b)\ast c)\ast d$.

Reduced nilpotent self-distributive algebras have most of the things that one would expect. Reduced nilpotent self-distributive algebras are often equipped with a composition operation. Reduced nilpotent self-distributive algebras have a notion of a critical point, and if our reduced nilpotent self-distributive algebra is endowed with a composition operation, the set of all critical points in the algebra forms a Heyting algebra. If $(X,\ast )$ is a self-distributive algebra, then we can define $x{\ast }_{0}y=y,x{\ast }_{n+1}y=x{\ast }_n(x\ast y)=x\ast \left(x{\ast }_{n}y\right)$ and$x{\ast }_{\infty }y={lim}_{n\to \infty }x{\ast }_{n}y$ in the discrete topology (this limit always exists for nilpotent self-distributive algebras). We define $x⪯y$ precisely when $x{\ast }_{\infty }y=1$ and $x\simeq y$ precisely when $x⪯y⪯x$. We can define $\text{crit}\left(x\right)$ to be the equivalence class of $x$ with respect to $\simeq$ and $\text{crit}\left[X\right]=X/\simeq$. The operations on $\text{crit}\left[X\right]$ are$\text{crit}\left(x\right)\to \text{crit}\left(y\right)=\text{crit}\left(x{\ast }_{\infty }y\right)$ and $\text{crit}\left(x\right)\wedge \text{crit}\left(y\right)=\text{crit}(x\circ y)$ whenever we have our composition operation $\circ$. One can use computer calculations to add another critical point to a reduced nilpotent self-distributive algebra and obtain a new reduced nilpotent self-distributive algebra with the same number of generators but with another critical point on top (and this new self-distributive algebra will be sub-directly irreducible). Therefore, by taking subdirect products and adding new critical points, we have an endless supply of reduced nilpotent self-distributive algebras with only 2 generators. I also know how to expand reduced nilpotent self-distributive algebras horizontally in the sense that given a finite reduced nilpotent self-distributive algebra $(X,\ast )$ that is not a Laver-like algebra, we can obtain another finite reduced nilpotent self-distributive algebra $(Y,\ast )$ where $X,Y$ are both generated by the same number of elements and these algebras both have the same implication algebra of critical points but $|Y|>>|X|$ but there is a surjective homomorphism $\varphi :Y\to X$. The point is that we have techniques for producing new nilpotent self-distributive algebras from old ones, and going deep into those new nilpotent self-distributive algebras.

Since reduced nilpotent self-distributive algebras are closed under finite subdirect products, subalgebras, and quotients, and since we have techniques for producing many nilpotent self-distributive algebras, perhaps one can make a ML model from these algebraic structures. On the other hand, there seems to be less of a connection between reduced nilpotent self-distributive algebras and large cardinals and non-commutative polynomials than with Laver-like algebras, so perhaps it is sufficient to stick to Laver-like algebras as a platform for constructing ML models.

From Laver-like algebras, we can also produce non-commutative polynomials. If $(X,\ast )$ is a Laver-like algebra, and $(x_a{)}_{a\in A}$ is a generating set for $X$, then for each non-maximal critical point $\alpha \in \text{crit}\left[X\right]$, we can define a non-commutative polynomial $p_{\alpha }\left(\right(x_a{)}_{a\in A})$ to be the sum of all non-commutative monomials of the form $x_{a_1}\dots x_{a_n}$ where $a_1\dots a_n\in A^{\ast }$ and $\text{crit}(a_1\ast \cdots \ast a_n)=\alpha$ but where $\text{crit}(a_1\ast \cdots \ast a_m)<\alpha$ for $1\le m<n$. These non-commutative polynomials capture all the information behind the Laver-like algebras since one can reconstruct the entire Laver-like algebra up to critical equivalence from these non-commutative polynomials. These non-commutative polynomials don't work as well for nilpotent self-distributive algebras; for nilpotent self-distributive algebras, these non-commutative polynomials will instead be rational expressions and one will not be able to recover the entire nilpotent self-distributive algebra from these rational expressions. 

I have used these non-commutative polynomials to construct the infinite product formula

$\dots p_r(x_1,\dots ,x_n)\dots p_0(x_1,\dots ,x_n)=\frac{1}{1-(x_1+\cdots +x_n)}$ where the polynomials $p_n(x_1,\dots ,x_n)$ are obtained from $n$ different non-trivial rank-into-rank embeddings $j_1,\dots ,j_n:V_{\lambda }\to V_{\lambda }$. This means that the non-commutative ring operations $+,\cdot$ in the rings of non-commutative polynomials are meaningful for Laver-like algebras.

If I wanted to interpret and make sense of a Laver-like algebra, I would use these non-commutative polynomials. Since non-commutative polynomials are closely related to strings (for NLP) and since the variables in these non-commutative polynomials can be substituted with matrices, I would not be surprised if one can turn Laver-like algebras into a full ML model using these non-commutative polynomials in order to solve a problem unrelated to Laver-like algebras.

Perhaps, one can also use strong large cardinal hypotheses and logic to produce ML models from Laver-like algebras. Since the existence of a non-trivial rank-into-rank embedding is among the strongest of all large cardinal hypotheses, and since one can produce useful theorems about natural looking finite algebraic structures from these large cardinal hypotheses, perhaps the interplay between the logic using large cardinal hypotheses and finite algebraic structures may be able to produce ML models? For example, one can automate the process of using elementarity to prove the existence of finite algebraic structures. After one has proven the existence of these structures using large cardinal hypotheses, one can do a search using backtracking in order to actually find candidates for these algebraic structures (or if the backtracking algorithm is exhaustive and turns up nothing, then we have a contradiction in the large cardinal hierarchy or a programming error. Mathematicians need to expend a lot of effort into finding an inconsistency in the large cardinal hierarchy this way because I am confident that they won't find such an inconsistency they will instead find plenty of near misses.). We can automate the process of producing examples of Laver-like algebras that satisfy conditions that were first established using large cardinal hypotheses, and we can perhaps completely describe these Laver-like algebras (up-to-critical equivalence) using our exhaustive backtracking process. This approach can be used to produce a lot of data about rank-into-rank embeddings, but do not see a clear path that allows us to apply this data outside of set theory and Laver-like algebras.

Using deep learning to investigate generalized Laver tables

We can certainly apply existing deep learning and machine learning techniques to classical and multigenic Laver tables.

The n-th classical Laver table is the unique self-distributive algebraic structure $A_n=\left(\right\{1,\dots ,2^n\},{\ast }_n)$ where$x{\ast }_{n}1=x+1\mathrm{mod}{2}^n$ whenever $x\in \{1,\dots ,2^n\}$. The classical Laver tables are up-to-isomorphism the only nilpotent self-distributive algebras generated by a single element.

Problem: Compute the $n$-th classical Laver table for as large $n$ as we can achieve.

Solution: Randall Dougherty in his 1995 paper Critical points in an algebra of elementary embeddings II gave an algorithm for computing the 48-th classical Laver table $A_{48}$ and with machine learning, I have personally gotten up to $A_{768}$ (I think I have a couple of errors, but those are not too significant), and I have some of the computation of $A_{1024}$.

Suppose now that $2^N\le n\le 3\cdot 2^N$. Then Dougherty has shown that one can easily recover the entire function ${\ast }_n$ from the restriction $L_{N,n}:\{1,\dots ,2^{2^N}\}\times \{1,\dots ,2^n\}\to \{1,\dots ,2^n\}$ where $L_{N,n}(x,y)=x{\ast }_{n}y$ for all $x,y$. Suppose now that $1\le x\le 2^n$. Then let $o_n\left(x\right)$ denote the least natural number $m$ such that $x{\ast }_n{2}^m=2^n$. Then there is a number ${\theta }_{n+1}\left(x\right)$ called the threshold of $x$ at $A_{n+1}$ such that $0\le {\theta }_{n+1}\left(x\right)\le 2^{o_n\left(x\right)}$ and ${\theta }_{n+1}\left(2^n\right)=0$ and such that for every $y$ with $1\le y\le 2^n$, we have $x{\ast }_{n+1}y=x{\ast }_{n}y$ whenever $1\le y\le {\theta }_{n+1}\left(x\right)$ and$x{\ast }_{n+1}y=\left(x{\ast }_{n}y\right)+2^n$ for ${\theta }_{n+1}\left(x\right)<y\le 2^{o_n\left(x\right)}$. The classical Laver table $A_{n+1}$ can be easily recovered from the table $A_n$ and the threshold function ${\theta }_{n+1}$. To go beyond Dougherty's calculation of $A_{48}$ to $A_{768}$ and beyond, we exploit a couple of observations about the threshold function ${\theta }_{n+1}$:

i: in most cases, ${\theta }_{n+2}\left(x\right)={\theta }_{n+1}\left(x\right)$, and

ii: in most cases, ${\theta }_{n+1}\left(x\right)=2^i-2^j$ for some $i,j$.

Let $E\left(n\right)=\{x\in \{1,\dots ,2^{n-2}\}:{\theta }_{n-1}(x)\ne {\theta }_n(x\left)\right\}$, and let $E^{♯}\left(n\right)=E\left(n\right)\cap \{1,\dots ,2^N-2\}$ where $N$ is the least natural number where $n\le 3\cdot 2^N$ and $T_n=\left\{\right(x,{\theta }_n\left(x\right)):x\in E^{♯}(n\left)\right\}$. Then one can easily compute $A_n$ from $A_{n-1}$ and the data $T_n$ by using Dougherty's algorithm. Now the only thing that we need to do is compute $T_n$ in the first place. It turns out that computing $T_n$ is quite easy except when $n$ is of the form $2^r+1$ or $3\cdot 2^r+1$ for some $r$. In the case, when $n=2^r+1$ or $n=3\cdot 2^r+1$, we initially set $T_n^{\text{temp}}=\varnothing .$ We then repeatedly modify $T_n^{\text{temp}}$ until we can no longer find any contradiction in the statement $T_n=T_n^{\text{temp}}$. Computing $T_n$ essentially amounts to finding new elements in the set $E^{♯}\left(n\right)$ and we find new elements in $E^{♯}\left(n\right)$ using some form of machine learning.

In my calculation of $A_{768}$, I used operations like bitwise AND, OR, and the bitwise majority operation to combine old elements in $E^{♯}\left(n\right)$ to find new elements in $E^{♯}\left(n\right)$, and I also employed other techniques similar to this. I did not use any neural networks though, but using neural networks seems to be a reasonable strategy, but we need to use the right kinds of neural networks.

My idea is to use something similar to a CNN where the linear layers are not as general as possible, but the linear layers are structured in a way that reduces the number of weights and so that the structure of the linear layers is compatible with the structure in the data. The elements in $E^{♯}\left(n\right)$ belong to $\{1,\dots ,2^N\}$ and $\{1,\dots ,2^N\}$ has a natural tensor product structure. One can therefore consider $E^{♯}\left(n\right)$ to be a subset of $(R^2{)}^{\otimes n}$. Furthermore, a tuple of $r$ elements in $E^{♯}\left(n\right)$ can be considered as a subset of $\left(R^2\right){)}^{\otimes n}\otimes R^r$. This means that the linear layers from $\left(R^2\right){)}^{\otimes n}\otimes R^r$ to $\left(R^2\right){)}^{\otimes n}\otimes R^r$ should be Kronecker products $A_1\otimes \cdots \otimes A_{n+1}$ or Kronecker sums $A_1\oplus \cdots \oplus A_{n+1}$ (it seems like Kronecker sums with residual layers would work better than Kronecker products). Recall that the Kronecker product and Kronecker sum of matrices $A,B$ are defined by setting $(A\otimes B)(u\otimes v)=Au\otimes Bv$ and $A\oplus B=(A\otimes 1)+(1\otimes B)$ respectively. Perhaps neural networks can be used to generate new elements in $E^{♯}\left(n\right)$. These generative neural networks do not have to be too large nor do they have to be too accurate. A neural network that is 50 percent accurate will be better than a neural network that is 90 percent accurate but takes 10 times as long to compute and is much harder to retrain once most of the elements in $E^{♯}\left(n\right)$ have already been obtained and when the neural network has trouble finding new elements. I also favor smaller neural networks because one can compute $A_{768}$ without using complicated techniques in the first place. I still need to figure out the rest of the details for the generative neural network that finds new elements of $E^{♯}\left(n\right)$ since I want it to be simple and easy to compute so that it is competitive with things like bitwise operations.

Problem: Translate Laver-like algebras into data (like sequences of vectors or matrices) into a form that machine learning models can work with.

Solution: The non-commutative polynomials $p_0,\dots ,p_n$ are already in a form that is easier for machine learning algorithms to use. I was able to apply an algorithm that I originally developed for analyzing block ciphers in order to turn the non-commutative polynomials $p_0,\dots ,p_n$ into unique sequences of real matrices $A_0,\dots ,A_n$ (these real matrices do not seem to depend on the initialization since the gradient ascent always seems to converge to the same local maximum). One can then feed this sequence of real matrices $A_0,\dots ,A_n$ into a transformer to solve whatever problem one wants to solve about Laver-like algebras. One can also represent a Laver-like algebra as a 2-dimensional image and then pass that image through a CNN to learn about the Laver-like algebra. This technique may only be used for Laver-like algebras that are small enough for CNNs to fully see, so I do not recommend CNNs for Laver-like algebras, but this is at least a proof-of-concept.

Problem: Estimate the number of small enough Laver-like algebras (up-to-critical equivalence) that satisfy certain properties.

Solution: The set of all multigenic Laver tables over an alphabet $A$ forms a rooted tree. We can therefore apply the technique for estimating the number of nodes in a rooted tree described in the 1975 paper Estimating the Efficiency of Backtrack Programs by Donald Knuth. This estimator is unbiased (the mean of the estimated value is actually the number of nodes in the tree), but it will have a very high variance for estimating the number of multigenic Laver tables. In order to reduce the variance for this estimator, we need to assign better probabilities when finding a random path from the root to the leaves, and we should be able to use transformers to assign these probabilities.

I am pretty sure that there are plenty of other ways to use machine learning to investigate Laver-like algebras.

Now, nilpotent self-distributive algebras may be applicable in cryptography (they seem like suitable platforms for the Kalka-Teicher key exchange but nobody has researched this). Perhaps a transformer that learns about Laver-like algebras would do better on some unrelated task than a transformer that was never trained using Laver-like algebras. But I do not see any other practical applications of nilpotent Laver-like algebras at the moment. This means that Laver-like algebras are currently a relatively safe problem for investigating machine learning algorithms, so Laver-like algebras are perhaps applicable to AI safety since I currently do not see any way of misaligning an AI for solving a problem related to Laver-like algebras.

The use of something like L1 regularization to achieve sparsity for inherent interpretability may just make things worse; a fixation on L1 regularization may lead people in the wrong direction. To avoid fixation, we should take a step back and look at the big picture. Occam's razor suggests that we should look for simple (and creative) solutions instead of over-engineering solutions when the entire foundation is inadequate. 

In order to obtain inherent interpretability, the machine learning model needs to behave in a way that is interesting to mathematicians. By piling on tweaks such as a lot of L1 or L0 regularization for sparsity, one is making the machine learning model more complicated. That makes it more difficult to study mathematically. And neural networks are inherently difficult to study mathematically and to interpret, so they should be replaced with something else. The problem is that neural networks already have so much momentum that people are unwilling to try anything else, and people are way too indoctrinated into neural networkology that they cannot learn new things. 

So how does one get momentum with a non-neural machine learning algorithm? One starts with shallow but mathematical machine learning algorithms first and one can also work with algorithms with few layers too. These shallow/few layer mathematical algorithms can still be effective for some problems since they have plenty of width. One may also construct a hybrid model where the first few layers are the mathematical construction but where the rest of the network is a deep neural network. I do not see how to make a very deep network this way, so the next steps are obscure to me.

I have not heard about the IBM paper until now. This is inspired by my personal experiments training (obviously classical) machine learning models.

Suppose that $V_0,\dots ,V_n,W_0,\dots ,W_n$ are real or complex finite dimensional inner product spaces. Suppose that the training data consists of tuples of the form $(v_0,\dots ,v_n)$ where $v_0\in V_0,\dots ,v_n\in V_n$ are vectors. Let $W_0=V_0$ and let $B_j:V_{j+1}\times W_j\to W_{j+1}$ be bilinear for all $j$. Then let

$L_v\left(w\right)=B_j(v,w)$ whenever $v\in V_{j+1},w\in W_j$. Then we define our polynomial by setting $p(v_0,\dots ,v_n)=L_{v_n}\dots L_{v_1}\left(v_0\right)$. In other words, my machine learning models are just compositions of bilinear mappings. In addition to wanting ${\text{Tr}}_U\left(\right[p(v_0,\dots ,v_n)\left]\right)$ to approximate the label, we also include regularization that makes the machine learning model pseudodeterministically trained so that if we train it twice with different initializations, we end up with the same trained model. Here, the machine learning model has $n$ layers, but the addition of extra layers gives us diminishing returns since bilinearity is close to linearity, so I still want to figure out how to improve the performance of such a machine learning model to match deep neural networks (if that is even feasible).

I use quantum information theory for my experiments mainly because quantum information theory behaves well unlike neural networks.

Proof-of-stake is still wasteful since it promotes pump and dump scams and causes people to waste their money on scam projects. If the creators are able to get their reward at the very beginning of a project, they will be more interested in short-term gains rather than a long-term token that will last. Humans are not psychologically/socially equipped to invest in proof-of-stake cryptocurrencies since they tend to get scammed.

Bitcoin mining is a real-world example of a goal that people spend an enormous amount of resources to attain, but this goal is useless or at least horribly inefficient. 

Recall that the orthogonality thesis states that it is possible for an intelligent entity to have bad or dumb goals and that it is also possible for a not-so-intelligent entity to have good goals. I would therefore consider Bitcoin mining to be a real-world prominent example of the orthogonality thesis as it in a sense a dumb goal attained intelligently (though, this example is imperfect).

Bitcoin's mining algorithm consists of computing many SHA-256 hashes relentlessly. The Bitcoin miners are rewarded whenever they compute a suitable SHA-256 hash that is lower than the target. These SHA-256 hashes establish decentralized consensus about the state of the blockchain, and they distribute newly minted bitcoins. But besides this, computing so many SHA-256 hashes is nearly useless. Computing so many SHA-256 hashes consumes large quantities of energy and creates electronic waste.

So what are some of the possible alternatives to Bitcoin mining? It seems like the best alternative that does not significantly change the nature of Bitcoin mining would be to replace SHA-256 mining with some other mining algorithm that serves some scientific purpose.

This is more difficult than it seems because Bitcoin mining must satisfy a list of cryptographic properties. If the mining algorithm did not satisfy these cryptographic problems, then it might not be feasible for newly minted bitcoins to be dispersed every 10 minutes, and we may enter a scenario where a single entity with a secret algorithm or slightly faster hardware were to put all the blocks on the blockchain.

Since Bitcoin mining must satisfy a list of cryptographic properties, it is difficult to come up with a more scientifically useful mining algorithm that satisfies these cryptographic properties. But in science, if there is a difficult problem, people should perform research on this scientific problem. While finding a useful cryptocurrency mining algorithm has its challenges, cryptocurrency mining algorithms are easy to produce since they can be made from cryptographic hash functions without requiring public key encryption or other advanced cryptographic algorithms, so difficulty seems more like an excuse rather than a legitimate reason not to investigate useful cryptocurrency mining algorithms. The cryptocurrency sector does not want to perform this research. I can think of several reasons why people refuse to support this sort of endeavor despite the great effort that people put into Bitcoin mining, but none of these reasons justify the lack of interest in useful cryptocurrency mining.

The diminishing quality of cryptocurrency users:

It seems like when altcoins were first being developed around 2014, people were much more interested in developing scientifically useful mining algorithms. But around 2017 when cryptocurrency really started to become popular, people simply wanted to make money from cryptocurrencies, yet they were not very interested in understanding how cryptocurrencies work or how to improve them.

Mining algorithms with questionable scientific use:

Some cryptocurrencies and proposals such as Primecoin and Gapcoin have more scientific mining algorithms, but these mining algorithms still have questionable usefulness. For example, the objective in Primecoin mining is to find suitable Cunningham chains. A Cunningham chain of the first kind is a sequence of prime numbers $(p_1,\dots ,p_n)$ where $p_{j+1}=2p_j+1$ whenever $1\le j<n$. The most interesting thing about Cunningham chains is that they can be used in cryptocurrency mining algorithms, but they are otherwise of minor importance to mathematics.

These questionable mining algorithms are supposed to steer the cryptocurrency community into a more scientific direction, but in reality, they have just steered the cryptocurrency community towards using mining to perform mathematical calculations that not even mathematicians care that much about.

Alternative solutions to the energy waste problem:

Many people just want to do away with cryptocurrency mining in an altcoin by replacing it with proof-of-stake or some other consensus mechanism. This solution is attractive to the cryptocurrency creators since they want complete control over all the coins at the beginning of the project, and they just use the energy usage of cryptocurrency as a marketing strategy to get people interested in their project. But this solution should not be appealing to anyone who wants to use the cryptocurrency even if a cryptocurrency is better funded without much mining (of course, if mining is replaced with another consensus mechanism after all the coins have been created, then this objection does not stand). After all, Satoshi Nakamoto did not fund Bitcoin by selling bitcoins. There are other ways to fund a cryptocurrency project without alternate consensus mechanisms.

Hostility against cryptocurrency technologies:

It seems like many members of society are hostile against cryptocurrency technologies and hate people who own or are in any way interested in cryptocurrency. This sort of hostility is a very good reason to conduct as many transactions using just cryptocurrency since I do not want to deal with all of those Karens. But this hostility may have turned people away from researching useful cryptocurrency mining algorithms even though the usefulness would probably not benefit the cryptocurrency directly.

Hardcore Bitcoiners:

If Bitcoin mining were magically replaced with a useful mining algorithm, barely anything about Bitcoin would change. But in my experience, Bitcoiners do not see it this way. They are so stuck in their ways that they reject all altcoins.

Conclusion:

While cryptocurrencies have a lot of monetary value, they are not exactly powerhouses of innovation, nor do I find them extremely interesting on their own. But a good scientific mining algorithm would make them much more innovative and interesting.

In this post, we shall go over a way to produce mostly linear machine learning classification models that output probabilities for each possible label. These mostly linear models are pseudodeterministically trained (or pseudodeterministic for short) in the sense that if we train them multiple times with different initializations, we will typically get the same trained model (up-to-symmetry and miniscule floating point differences).

The algorithms that I am mentioning in this post generalize to more complicated multi-layered algorithms in the sense that the multi-layered algorithms remain pseudodeterministic, but for simplicity, we shall stick to just linear operators here.

Let $K$ denote either the field of real numbers, the field of complex numbers, or the division ring of quaternions. Let $U$ be a finite dimensional inner product space over $K$. The training data is a set $D$ of pairs $(u,v)$ where $u\in U$ and $v\in \{1,\dots ,n\}$ where $u$ is the machine learning model input and $v$ is the label. The machine learning model is trained to predict the label $v$ when given the input $u$. The trained model is a function $f$ that maps $U$ to the set of all probability vectors of length $n$, so the trained model actually gives the probabilities for each possible label.

Suppose that $V_i$ is a finite dimensional inner product space over $K$ for each $i\in \{1,\dots ,n\}$. Then the domain of the fitness function consists of tuples $(A_1,\dots ,A_n)$ where each $A_i$ is a linear operator from $U$ to $V_i$. Let $p\in (0,1),r\in (0,\infty ),q\in (1,\infty )$, and let $\lambda \ge 0$. The parameter $p$ is the exponent while $\lambda$ is the regularization parameter. Define (almost total) functions $G,R,F:L(U,V_1)\times \cdots \times L(U,V_n)\to R$ by setting

$G(A_1,\dots ,A_n)={\sum }_{(u,v)\in D}(\frac{\parallel A_{v}u{\parallel }^r}{\parallel A_{1}u{\parallel }^r+\cdots +\parallel A_{n}u{\parallel }^r}{)}^p/|D|$

$R(A_1,\dots ,A_n)=({\sum }_{(u,v)\in D}\lambda \cdot \log (\parallel A_{v}u\parallel )/|D|)$

$-\lambda \cdot \left(\log \right(\parallel A_1{\parallel }_q)+\cdots +\log (\parallel A_n{\parallel }_q\left)\right)/n$.

Here, $\parallel \ast {\parallel }_q$ denotes the Schatten $q$-norm which can be defined by setting

$\parallel A{\parallel }_q=\text{Tr}\left(\right(A{A}^{\ast }{)}^{q/2})$.

Set $F=G+R$. Here, $F$ denotes our fitness function. The function $G$ what we really want to maximize, but unfortunately, $G$ is typically non-pseudodeterministic, so we need to add the regularization term $R$ to obtain pseudodeterminism. The regularization term $R$ also has the added effect of making $\parallel A_{v}u\parallel$ relatively large compared to the norm $\parallel A{\parallel }_q$ for training data points $(u,v)$. This may be useful in determining whether a pair should belong to either the training or test data in the first place.

We observe that $F$ is $0$-homogeneous in the sense that$F(A_1,\dots ,A_n)=F(c{A}_1,\dots ,c{A}_n)$ for each non-zero scalar $c$ (in the quaternionic case, the scalars are just the real numbers).

Suppose now that we have obtained a tuple $(A_1,\dots ,A_n)$ that maximizes the fitness $F(A_1,\dots ,A_n)$. Let $PV\left(n\right)$ denote the set of all probability vectors of length $n$. Then define an almost total function $f:U\to PV\left(n\right)$ by setting

$f\left(u\right)=\frac{(\parallel A_{1}u{\parallel }^{r(1-p)},\dots ,\parallel A_{n}u{\parallel }^{r(1-p)})}{\parallel A_{1}u{\parallel }^{r(1-p)}+\cdots +\parallel A_{n}u{\parallel }^{r(1-p)}}.$

If $(u,v)$ belongs to the training data set, then the $i$-th entry of $f\left(u\right)$ is the machine learning model's estimate of the probability that $i=v$. I will let the reader justify this calculation of the probabilities.

We can generalize the function $f$ to pseudodeterministically trained machine learning models with multiple layers by replacing the linear operators $A_1,\dots ,A_n$with some non-linear or multi-linear operators. Actually, there are quite a few ways of generalizing the fitness function $F$, and I have taken some liberty in the exact formulation for $F$.

In addition to being pseudodeterministic, the fitness function $F$ has other notable desirable properties. For example, when maximizing $F$ using gradient ascent, one tends to converge to the local maximum at an exponential rate without needing to decay the learning rate.