The intersection of machine learning and generalizations of Laver tables seems to have a lot of potential, so your question about this is extremely interesting to me.

Machine learning models from generalized Laver tables?

I have not been able to find any machine learning models that are based on generalized Laver tables that are used for solving problems unrelated to Laver tables. I currently do not directly see any new kinds of deep learning models that one can produce from generalizations of Laver tables, but I would not be surprised if there were a non-standard machine learning model from Laver like algebras that we could use for something like NLP. 

But I have to be skeptical about applying machine learning to generalized Laver tables. Generalizations of Laver tables have intricate structure and complexity but this complexity is ordered. Furthermore, there is a very large supply of Laver-like algebras (even with 2 generators) for potentially any situation. Laver-like algebras are also very easy to compute. While backtracking may potentially take an exponential amount of time, when I have computed Laver-like algebras, the backtracking algorithm seems to always terminate quickly unless it is producing a large quantity or Laver-like algebras or larger Laver-like algebras. But with all this being said, Laver-like algebras seem to lack the steerability that we need to apply these algebras to solving real-world problems. For example, try interpreting (in a model theoretic sense) modular arithmetic modulo 13 in a Laver-like algebra. That is quite hard to do because Laver-like algebras are not built for that sort of thing.

Here are a couple of avenues that I see as most promising for producing new machine learning algorithms from Laver-like algebras and other structures.

Let $X$ be a set, and let $\ast$ be a binary operation on $X$ that satisfies the self-distributivity identity $x\ast (y\ast z)=(x\ast y)\ast (x\ast z)$. Define the right powers $x^{\left[n\right]}$ for $n\ge 1$ inductively by setting $x^{\left[1\right]}=x$ and $x^{[n+1]}=x\ast x^{\left[n\right]}$. We say that $(X,\ast ,1)$ is a reduced nilpotent self-distributive algebra if $1$ satisfies the identities $1\ast x=x,x\ast 1=1$ for $x\in X$ and if for all $x\in X$ there is an $n\ge 1$ with $x^{\left[n\right]}=1$. A reduced Laver-like algebra is a reduced nilpotent self-distributive algebra $(X,\ast )$ where if $x_n\in X$ for each $n\ge 0$, then $x_0\ast \cdots \ast x_N=1$ for some $N$. Here we make the convention to put the implied parentheses on the left so that $a\ast b\ast c\ast d=\left(\right(a\ast b)\ast c)\ast d$.

Reduced nilpotent self-distributive algebras have most of the things that one would expect. Reduced nilpotent self-distributive algebras are often equipped with a composition operation. Reduced nilpotent self-distributive algebras have a notion of a critical point, and if our reduced nilpotent self-distributive algebra is endowed with a composition operation, the set of all critical points in the algebra forms a Heyting algebra. If $(X,\ast )$ is a self-distributive algebra, then we can define $x{\ast }_{0}y=y,x{\ast }_{n+1}y=x{\ast }_n(x\ast y)=x\ast \left(x{\ast }_{n}y\right)$ and$x{\ast }_{\infty }y={lim}_{n\to \infty }x{\ast }_{n}y$ in the discrete topology (this limit always exists for nilpotent self-distributive algebras). We define $x⪯y$ precisely when $x{\ast }_{\infty }y=1$ and $x\simeq y$ precisely when $x⪯y⪯x$. We can define $\text{crit}\left(x\right)$ to be the equivalence class of $x$ with respect to $\simeq$ and $\text{crit}\left[X\right]=X/\simeq$. The operations on $\text{crit}\left[X\right]$ are$\text{crit}\left(x\right)\to \text{crit}\left(y\right)=\text{crit}\left(x{\ast }_{\infty }y\right)$ and $\text{crit}\left(x\right)\wedge \text{crit}\left(y\right)=\text{crit}(x\circ y)$ whenever we have our composition operation $\circ$. One can use computer calculations to add another critical point to a reduced nilpotent self-distributive algebra and obtain a new reduced nilpotent self-distributive algebra with the same number of generators but with another critical point on top (and this new self-distributive algebra will be sub-directly irreducible). Therefore, by taking subdirect products and adding new critical points, we have an endless supply of reduced nilpotent self-distributive algebras with only 2 generators. I also know how to expand reduced nilpotent self-distributive algebras horizontally in the sense that given a finite reduced nilpotent self-distributive algebra $(X,\ast )$ that is not a Laver-like algebra, we can obtain another finite reduced nilpotent self-distributive algebra $(Y,\ast )$ where $X,Y$ are both generated by the same number of elements and these algebras both have the same implication algebra of critical points but $|Y|>>|X|$ but there is a surjective homomorphism $\varphi :Y\to X$. The point is that we have techniques for producing new nilpotent self-distributive algebras from old ones, and going deep into those new nilpotent self-distributive algebras.

Since reduced nilpotent self-distributive algebras are closed under finite subdirect products, subalgebras, and quotients, and since we have techniques for producing many nilpotent self-distributive algebras, perhaps one can make a ML model from these algebraic structures. On the other hand, there seems to be less of a connection between reduced nilpotent self-distributive algebras and large cardinals and non-commutative polynomials than with Laver-like algebras, so perhaps it is sufficient to stick to Laver-like algebras as a platform for constructing ML models.

From Laver-like algebras, we can also produce non-commutative polynomials. If $(X,\ast )$ is a Laver-like algebra, and $(x_a{)}_{a\in A}$ is a generating set for $X$, then for each non-maximal critical point $\alpha \in \text{crit}\left[X\right]$, we can define a non-commutative polynomial $p_{\alpha }\left(\right(x_a{)}_{a\in A})$ to be the sum of all non-commutative monomials of the form $x_{a_1}\dots x_{a_n}$ where $a_1\dots a_n\in A^{\ast }$ and $\text{crit}(a_1\ast \cdots \ast a_n)=\alpha$ but where $\text{crit}(a_1\ast \cdots \ast a_m)<\alpha$ for $1\le m<n$. These non-commutative polynomials capture all the information behind the Laver-like algebras since one can reconstruct the entire Laver-like algebra up to critical equivalence from these non-commutative polynomials. These non-commutative polynomials don't work as well for nilpotent self-distributive algebras; for nilpotent self-distributive algebras, these non-commutative polynomials will instead be rational expressions and one will not be able to recover the entire nilpotent self-distributive algebra from these rational expressions. 

I have used these non-commutative polynomials to construct the infinite product formula

$\dots p_r(x_1,\dots ,x_n)\dots p_0(x_1,\dots ,x_n)=\frac{1}{1-(x_1+\cdots +x_n)}$ where the polynomials $p_n(x_1,\dots ,x_n)$ are obtained from $n$ different non-trivial rank-into-rank embeddings $j_1,\dots ,j_n:V_{\lambda }\to V_{\lambda }$. This means that the non-commutative ring operations $+,\cdot$ in the rings of non-commutative polynomials are meaningful for Laver-like algebras.

If I wanted to interpret and make sense of a Laver-like algebra, I would use these non-commutative polynomials. Since non-commutative polynomials are closely related to strings (for NLP) and since the variables in these non-commutative polynomials can be substituted with matrices, I would not be surprised if one can turn Laver-like algebras into a full ML model using these non-commutative polynomials in order to solve a problem unrelated to Laver-like algebras.

Perhaps, one can also use strong large cardinal hypotheses and logic to produce ML models from Laver-like algebras. Since the existence of a non-trivial rank-into-rank embedding is among the strongest of all large cardinal hypotheses, and since one can produce useful theorems about natural looking finite algebraic structures from these large cardinal hypotheses, perhaps the interplay between the logic using large cardinal hypotheses and finite algebraic structures may be able to produce ML models? For example, one can automate the process of using elementarity to prove the existence of finite algebraic structures. After one has proven the existence of these structures using large cardinal hypotheses, one can do a search using backtracking in order to actually find candidates for these algebraic structures (or if the backtracking algorithm is exhaustive and turns up nothing, then we have a contradiction in the large cardinal hierarchy or a programming error. Mathematicians need to expend a lot of effort into finding an inconsistency in the large cardinal hierarchy this way because I am confident that they won't find such an inconsistency they will instead find plenty of near misses.). We can automate the process of producing examples of Laver-like algebras that satisfy conditions that were first established using large cardinal hypotheses, and we can perhaps completely describe these Laver-like algebras (up-to-critical equivalence) using our exhaustive backtracking process. This approach can be used to produce a lot of data about rank-into-rank embeddings, but do not see a clear path that allows us to apply this data outside of set theory and Laver-like algebras.

Using deep learning to investigate generalized Laver tables

We can certainly apply existing deep learning and machine learning techniques to classical and multigenic Laver tables.

The n-th classical Laver table is the unique self-distributive algebraic structure $A_n=\left(\right\{1,\dots ,2^n\},{\ast }_n)$ where$x{\ast }_{n}1=x+1\mathrm{mod}{2}^n$ whenever $x\in \{1,\dots ,2^n\}$. The classical Laver tables are up-to-isomorphism the only nilpotent self-distributive algebras generated by a single element.

Problem: Compute the $n$-th classical Laver table for as large $n$ as we can achieve.

Solution: Randall Dougherty in his 1995 paper Critical points in an algebra of elementary embeddings II gave an algorithm for computing the 48-th classical Laver table $A_{48}$ and with machine learning, I have personally gotten up to $A_{768}$ (I think I have a couple of errors, but those are not too significant), and I have some of the computation of $A_{1024}$.

Suppose now that $2^N\le n\le 3\cdot 2^N$. Then Dougherty has shown that one can easily recover the entire function ${\ast }_n$ from the restriction $L_{N,n}:\{1,\dots ,2^{2^N}\}\times \{1,\dots ,2^n\}\to \{1,\dots ,2^n\}$ where $L_{N,n}(x,y)=x{\ast }_{n}y$ for all $x,y$. Suppose now that $1\le x\le 2^n$. Then let $o_n\left(x\right)$ denote the least natural number $m$ such that $x{\ast }_n{2}^m=2^n$. Then there is a number ${\theta }_{n+1}\left(x\right)$ called the threshold of $x$ at $A_{n+1}$ such that $0\le {\theta }_{n+1}\left(x\right)\le 2^{o_n\left(x\right)}$ and ${\theta }_{n+1}\left(2^n\right)=0$ and such that for every $y$ with $1\le y\le 2^n$, we have $x{\ast }_{n+1}y=x{\ast }_{n}y$ whenever $1\le y\le {\theta }_{n+1}\left(x\right)$ and$x{\ast }_{n+1}y=\left(x{\ast }_{n}y\right)+2^n$ for ${\theta }_{n+1}\left(x\right)<y\le 2^{o_n\left(x\right)}$. The classical Laver table $A_{n+1}$ can be easily recovered from the table $A_n$ and the threshold function ${\theta }_{n+1}$. To go beyond Dougherty's calculation of $A_{48}$ to $A_{768}$ and beyond, we exploit a couple of observations about the threshold function ${\theta }_{n+1}$:

i: in most cases, ${\theta }_{n+2}\left(x\right)={\theta }_{n+1}\left(x\right)$, and

ii: in most cases, ${\theta }_{n+1}\left(x\right)=2^i-2^j$ for some $i,j$.

Let $E\left(n\right)=\{x\in \{1,\dots ,2^{n-2}\}:{\theta }_{n-1}(x)\ne {\theta }_n(x\left)\right\}$, and let $E^{♯}\left(n\right)=E\left(n\right)\cap \{1,\dots ,2^N-2\}$ where $N$ is the least natural number where $n\le 3\cdot 2^N$ and $T_n=\left\{\right(x,{\theta }_n\left(x\right)):x\in E^{♯}(n\left)\right\}$. Then one can easily compute $A_n$ from $A_{n-1}$ and the data $T_n$ by using Dougherty's algorithm. Now the only thing that we need to do is compute $T_n$ in the first place. It turns out that computing $T_n$ is quite easy except when $n$ is of the form $2^r+1$ or $3\cdot 2^r+1$ for some $r$. In the case, when $n=2^r+1$ or $n=3\cdot 2^r+1$, we initially set $T_n^{\text{temp}}=\varnothing .$ We then repeatedly modify $T_n^{\text{temp}}$ until we can no longer find any contradiction in the statement $T_n=T_n^{\text{temp}}$. Computing $T_n$ essentially amounts to finding new elements in the set $E^{♯}\left(n\right)$ and we find new elements in $E^{♯}\left(n\right)$ using some form of machine learning.

In my calculation of $A_{768}$, I used operations like bitwise AND, OR, and the bitwise majority operation to combine old elements in $E^{♯}\left(n\right)$ to find new elements in $E^{♯}\left(n\right)$, and I also employed other techniques similar to this. I did not use any neural networks though, but using neural networks seems to be a reasonable strategy, but we need to use the right kinds of neural networks.

My idea is to use something similar to a CNN where the linear layers are not as general as possible, but the linear layers are structured in a way that reduces the number of weights and so that the structure of the linear layers is compatible with the structure in the data. The elements in $E^{♯}\left(n\right)$ belong to $\{1,\dots ,2^N\}$ and $\{1,\dots ,2^N\}$ has a natural tensor product structure. One can therefore consider $E^{♯}\left(n\right)$ to be a subset of $(R^2{)}^{\otimes n}$. Furthermore, a tuple of $r$ elements in $E^{♯}\left(n\right)$ can be considered as a subset of $\left(R^2\right){)}^{\otimes n}\otimes R^r$. This means that the linear layers from $\left(R^2\right){)}^{\otimes n}\otimes R^r$ to $\left(R^2\right){)}^{\otimes n}\otimes R^r$ should be Kronecker products $A_1\otimes \cdots \otimes A_{n+1}$ or Kronecker sums $A_1\oplus \cdots \oplus A_{n+1}$ (it seems like Kronecker sums with residual layers would work better than Kronecker products). Recall that the Kronecker product and Kronecker sum of matrices $A,B$ are defined by setting $(A\otimes B)(u\otimes v)=Au\otimes Bv$ and $A\oplus B=(A\otimes 1)+(1\otimes B)$ respectively. Perhaps neural networks can be used to generate new elements in $E^{♯}\left(n\right)$. These generative neural networks do not have to be too large nor do they have to be too accurate. A neural network that is 50 percent accurate will be better than a neural network that is 90 percent accurate but takes 10 times as long to compute and is much harder to retrain once most of the elements in $E^{♯}\left(n\right)$ have already been obtained and when the neural network has trouble finding new elements. I also favor smaller neural networks because one can compute $A_{768}$ without using complicated techniques in the first place. I still need to figure out the rest of the details for the generative neural network that finds new elements of $E^{♯}\left(n\right)$ since I want it to be simple and easy to compute so that it is competitive with things like bitwise operations.

Problem: Translate Laver-like algebras into data (like sequences of vectors or matrices) into a form that machine learning models can work with.

Solution: The non-commutative polynomials $p_0,\dots ,p_n$ are already in a form that is easier for machine learning algorithms to use. I was able to apply an algorithm that I originally developed for analyzing block ciphers in order to turn the non-commutative polynomials $p_0,\dots ,p_n$ into unique sequences of real matrices $A_0,\dots ,A_n$ (these real matrices do not seem to depend on the initialization since the gradient ascent always seems to converge to the same local maximum). One can then feed this sequence of real matrices $A_0,\dots ,A_n$ into a transformer to solve whatever problem one wants to solve about Laver-like algebras. One can also represent a Laver-like algebra as a 2-dimensional image and then pass that image through a CNN to learn about the Laver-like algebra. This technique may only be used for Laver-like algebras that are small enough for CNNs to fully see, so I do not recommend CNNs for Laver-like algebras, but this is at least a proof-of-concept.

Problem: Estimate the number of small enough Laver-like algebras (up-to-critical equivalence) that satisfy certain properties.

Solution: The set of all multigenic Laver tables over an alphabet $A$ forms a rooted tree. We can therefore apply the technique for estimating the number of nodes in a rooted tree described in the 1975 paper Estimating the Efficiency of Backtrack Programs by Donald Knuth. This estimator is unbiased (the mean of the estimated value is actually the number of nodes in the tree), but it will have a very high variance for estimating the number of multigenic Laver tables. In order to reduce the variance for this estimator, we need to assign better probabilities when finding a random path from the root to the leaves, and we should be able to use transformers to assign these probabilities.

I am pretty sure that there are plenty of other ways to use machine learning to investigate Laver-like algebras.

Now, nilpotent self-distributive algebras may be applicable in cryptography (they seem like suitable platforms for the Kalka-Teicher key exchange but nobody has researched this). Perhaps a transformer that learns about Laver-like algebras would do better on some unrelated task than a transformer that was never trained using Laver-like algebras. But I do not see any other practical applications of nilpotent Laver-like algebras at the moment. This means that Laver-like algebras are currently a relatively safe problem for investigating machine learning algorithms, so Laver-like algebras are perhaps applicable to AI safety since I currently do not see any way of misaligning an AI for solving a problem related to Laver-like algebras.

Whether one takes or should take a cold shower or not depends on a lot of factors including whether one exercises, one's health, one's personal preferences, the air temperature, the cold water temperature, the humidity level, and the hardness of the shower water. But it seems like most people can't fathom taking a cold shower simply because they are cold intolerant even though cold showers have many benefits.

In addition to the practical benefits of cold showers, cold showers also may offer health benefits.

Cold showers could improve one's immune system (though we should). 

The Effect of Cold Showering on Health and Work: A Randomized Controlled Trial - PMC

Cold showers may boost mood or alleviate depression.

Scientific Evidence-Based Effects of Hydrotherapy on Various Systems of the Body - PMC

Adapted cold shower as a potential treatment for depression - ScienceDirect

Cold showers could also improve circulation and metabolism.

Cold showers also offer other benefits.

I always use the exhaust fan. It is never powerful enough to reduce the humidity faster than a warm shower increases the humidity. I also lock the door when taking a shower, and I do not know why anyone would take a shower without locking the door. Opening the door while showering just makes the rest of the home humid as well, and we can't have that.

I exercise daily, so out of habit, I always take a shower after I exercise, and most of my showers are after exercise. Even if I spend a few minutes cooling down after exercise, I need the shower to cool down even more, and by taking a warm shower, I cannot cool down as effectively, so I end up sweating after taking the shower. And I sometimes take my temperature after exercise and the shower and even after the shower, I tend to have a mouth temperature of 99.0 to 99.5 degrees Fahrenheit. I doubt that people who barely need to take a shower after exercising are doing much exercise or perhaps they are doing weights instead of cardio which produces less sweat, but in any case, I have never exercised and thought that I do not need a shower regardless of whether I am doing cardio, weights, or whatever.

Soap scum left over after taking a cold shower seems to be a problem for you and for you only. 

Added 8/20/2025: And taking a hot shower produces all the condensation that helps all that mirror bacteria grow. Biological risk from the mirror world — LessWrong

 I am going to share an algorithm that I came up with that tends to produce the same result when we run it multiple times with a different initialization. The iteration is not even guaranteed convergence since we are not using gradient ascent, but it typically converges as long as the algorithm is given a reasonable input. This suggests that the algorithm behaves mathematically and may be useful for things such as quantum error correction. After analyzing the algorithm, I shall use the algorithm to solve a computational problem.

We say that an algorithm is pseudodeterministic if it tends to return the same output even if the computation leading to that output is non-deterministic (due to a random initialization). I believe that we should focus a lot more on pseudodetermistic machine learning algorithms for AI safety and interpretability since pseudodeterministic algorithms are inherently interpretable.

Define $f\left(z\right)=3z^2-2z^3$ for all complex numbers $z$. Then $f\left(0\right)=0,f\left(1\right)=1,f^{\prime }\left(0\right)=f^{\prime }\left(1\right)=0$, and there are neighborhoods $U,V$ of $0,1$ respectively where if $x\in U$, then $f^N\left(x\right)\to 0$ quickly and if $y\in V$, then $f^N\left(y\right)\to 1$ quickly. Set $f^{\infty }={lim}_{N\to \infty }{f}^N$. The function $f^{\infty }$ serves as error correction for projection matrices since if $Q$ is nearly a projection matrix, then $f^{\infty }\left(Q\right)$ will be a projection matrix.

Suppose that $K$ is either the field of real numbers, complex numbers or quaternions. Let $Z\left(K\right)$ denote the center of $K$. In particular, $Z\left(R\right)=R,Z\left(C\right)=C,Z\left(H\right)=R$. 

If $A_1,\dots ,A_r$ are $m\times n$-matrices, then define $\Phi (A_1,\dots ,A_r):M_n\left(K\right)\to M_m\left(K\right)$ by setting $\Phi (A_1,\dots ,A_r)={\sum }_{k=1}^r{A}_{k}X{A}_k^{\ast }$. Then we say that an operator of the form $\Phi (A_1,\dots ,A_r)$ is completely positive. We say that a $Z\left(K\right)$-linear operator $E:M_n\left(K\right)\to M_m\left(K\right)$ is Hermitian preserving if $E\left(X\right)$ is Hermitian whenever $X$ is Hermitian. Every completely positive operator is Hermitian preserving.

Suppose that $E:M_n\left(K\right)\to M_n\left(K\right)$ is $Z\left(K\right)$-linear. Let $t>0$. Let $P_0\in M_n\left(K\right)$ be a random orthogonal projection matrix of rank $d$. Set $P_{N+1}=f^{\infty }(P_N+t\cdot E(P_N\left)\right)$ for all $N$. Then if everything goes well, the sequence $(P_N{)}_N$ will converge to a projection matrix $P$ of rank $d$, and the projection matrix $P$ will typically be unique in the sense that if we run the experiment again, we will typically obtain the exact same projection matrix $P$. If $E$ is Hermitian preserving, then the projection matrix $P$ will typically be an orthogonal projection. This experiment performs well especially when $E$ is completely positive or at least Hermitian preserving or nearly so. The projection matrix $P$ will satisfy the equation $P\cdot E\left(P\right)=E\left(P\right)\cdot P=P\cdot E\left(P\right)\cdot P$.

In the case when $E$ is a quantum channel, we can easily explain what the projection $P$ does. The operator $P$ is a projection onto a subspace of complex Euclidean space that is particularly well preserved by the channel $E$. In particular, the image $\text{Im}\left(P\right)$ is spanned by the top $d$ eigenvectors of $E\left(P\right)$. This means that if we send the completely mixed state $P/d$ through the quantum channel $E$ and we measure the state $E\left(P/d\right)$ with respect to the projective measurement $(P,I-P)$, then there is an unusually high probability that this measurement will land on $P$ instead of $I-P$.

Let us now use the algorithm that obtains $P$ from $E$ to solve a problem in many cases.

If $x$ is a vector, then let $\text{Diag}\left(x\right)$ denote the diagonal matrix where $x$ is the vector of diagonal entries, and if $X$ is a square matrix, then let $\text{Diag}\left(X\right)$ denote the diagonal of $X$. If $x$ is a length $n$ vector, then $\text{Diag}\left(x\right)$ is an $n\times n$-matrix, and if $X$ is an $n\times n$-matrix, then $\text{Diag}\left(X\right)$ is a length $n$ vector.

Problem Input: An $n\times n$-square matrix $A$ with non-negative real entries and a natural number $d$ with $1\le d<n$.

Objective: Find a subset $B\subseteq \{1,\dots ,n\}$ with $|B|=d$ and where if $x=A\cdot {\chi }_B$, then the $d$ largest entries in $x$ are the values $x\left[b\right]$ for $b\in B$.

Algorithm: Let $E$ be the completely positive operator defined by setting $E\left(X\right)=\text{Diag}(A\cdot \text{Diag}(X\left)\right)$. Then we run the iteration using $E$ to produce an orthogonal projection $P$ with rank $d$. In this case, the projection $P$ will be a diagonal projection matrix with rank $d$ where $\text{diag}\left(P\right)={\chi }_B$ and where $B$ is our desired subset of $\{1,\dots ,n\}$.

While the operator $P$ is just a linear operator, the pseudodeterminism of the algorithm that produces the operator $P$ generalizes to other pseudodeterministic algorithms that return models that are more like deep neural networks.

I would have thought that a fitness function that is maximized using something other than gradient ascent and which can solve NP-complete problems at least in the average case would be worth reading since that means that it can perform well on some tasks but it also behaves mathematically in a way that is needed for interpretability. The quality of the content is inversely proportional to the number of views since people don't think the same way as I do.

Wheels on the Bus | @CoComelon Nursery Rhymes & Kids Songs

Stuff that is popular is usually garbage.

But here is my post about the word embedding.

Interpreting a matrix-valued word embedding with a mathematically proven characterization of all optima — LessWrong

And I really do not want to collaborate with people who are not willing to read the post. This is especially true of people in academia since universities promote violence and refuse to acknowledge any wrongdoing. Universities are the absolute worst.

Instead of engaging with the actual topic, people tend to just criticize stupid stuff simply because they only want to read about what they already know or what is recommended by their buddies; that is a very good way not to learn anything new or insightful. For this reason, even the simplest concepts are lost on most people.

In this post, the existence of a non-gradient based algorithm for computing LSRDRs is a sign that LSRDRs behave mathematically and are quite interpretable. Gradient ascent is a general purpose optimization algorithm that works in the case when there is no other way to solve the optimization problem, but when there are multiple ways of obtaining a solution to an optimization problem, the optimization problem is behaving in a way that should be appealing to mathematicians.

LSRDRs and similar algorithms are pseudodeterministic in the sense that if we train the model multiple times on the same data, we typically get identical models. Pseudodeterminism is a signal of interpretability for several reasons that I will go into more detail in a future post:

1. Pseudodeterministic models do not contain any extra random or even pseudorandom information that is not contained in the training data already. This means that when interpreting these models, one does not have to interpret random information.
2. Pseudodeterministic models inherit the symmetry of their training data. For example, if we train a real LSRDR using real symmetric matrices, then the projection $P$ will itself by a symmetric matrix.
3. In mathematics, a well-posed problem is a problem where there exists a unique solution to the problem. Well-posed problems behave better than ill-posed problems in the sense that it is easier to prove results about well-posed problems than it is to prove results about ill-posed problems.

In addition to pseudodeterminism, in my experience, LSRDRs are quite interpretable since I have interpreted LSRDRs already in a few posts:

Interpreting a dimensionality reduction of a collection of matrices as two positive semidefinite block diagonal matrices — LessWrong

When performing a dimensionality reduction on tensors, the trace is often zero. — LessWrong

I have Generalized LSRDRs so that they are starting to behave like deeper neural networks. I am trying to expand the capabilities of generalized LSRDRs so they behave more like deep neural networks, but I still have some work to expand their capabilities while retaining pseudodeterminism. In the meantime, generalized LSRDRs may still function as narrow AI for specific problems and also as layers in AI.

Of course, if we want to compare capabilities, we should also compare NNs to LSRDRs at tasks such as evaluating the cryptographic security of block ciphers, solving NP-complete problems in the average case, etc.

As for the difficulty of this post, it seems like that is the result of the post being mathematical. But going through this kind of mathematics so that we obtain inherently interpretable AI should be the easier portion of AI interpretability. I would much rather communicate about the actual mathematics than about how difficult the mathematics is.

In this post, we shall describe 3 related fitness functions with discrete domains where the process of maximizing these functions is pseudodeterministic in the sense that if we locally maximize the fitness function multiple times, then we typically attain the same local maximum; this appears to be an important aspect of AI safety. These fitness functions are my own. While these functions are far from deep neural networks, I think they are still related to AI safety since they are closely related to other fitness functions that are locally maximized pseudodeterministically that more closely resemble deep neural networks.

Let $K$ denote a finite dimensional algebra over the field of real numbers together with an adjoint operation $\ast$ (the operation $\ast$ is a linear involution with $(xy{)}^{\ast }=y^{\ast }{x}^{\ast }$). For example, $K$ could be the field of real numbers, complex numbers, quaternions, or a matrix ring over the reals, complex, or quaternions. We can extend the adjoint $\ast$ to the matrix ring $M_r\left(K\right)$ by setting $(x_{i,j}{)}_{i,j}^{\ast }=(x_{j,i}^{\ast }{)}_{i,j}$.

Let $n$ be a natural number. If $A_1,\dots ,A_r\in M_n\left(K\right),X_1,\dots ,X_r\in M_d\left(K\right)$, then define

$\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r):M_{n,d}\left(K\right)\to M_{n,d}\left(K\right)$ by setting $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r)\left(X\right)=A_{1}X{X}_1^{\ast }+\cdots +A_{r}X{X}_r^{\ast }$.

Suppose now that $1\le d<n$. Then let $S_d\subseteq M_{n,n}\left(K\right)$ be the set of all $0,1$-diagonal matrices with $d$ many $1$'s on the diagonal. We observe that each element in $S_d$ is an orthogonal projection. Define fitness functions $F_d,G_d,H_d:S_d\to R$ by setting

$F_d\left(P\right)=\rho \left(\Gamma \right(A_1,\dots ,A_r;P{A}_{1}P,\dots ,P{A}_{r}P\left)\right)$,

$G_d\left(P\right)=\rho \left(\Gamma \right(P{A}_{1}P,\dots ,P{A}_{r}P;P{A}_{1}P,\dots ,P{A}_{r}P\left)\right)$, and

$H_d\left(P\right)=\frac{F_d(P{)}^2}{G_d\left(P\right)}$. Here, $\rho$ denotes the spectral radius.

$F_d\left(P\right)$ is typically slightly larger than $G_d\left(P\right)$, so these three fitness functions are closely related.

If $P,Q\in S_d$, then we say that $Q$ is in the neighborhood of $P$ if $Q$ differs from $P$ by at most 2 entries. If $F$ is a fitness function with domain $S_d$, then we say that $(P,F(P\left)\right)$ is a local maximum of the function $F$ if $F\left(P\right)\ge F\left(Q\right)$ whenever $Q$ is in the neighborhood of $P$. 

The path from initialization to a local maximum  $(P_s,F(P_s\left)\right)$ for will be a sequence $(P_0,\dots ,P_s)$ where $P_j$ is always in the neighborhood of $P_{j-1}$ and where $F\left(P_j\right)\ge F\left(P_{j-1}\right)$ for all $j$ and the length of the path will be $s$ and where $P_0$ is generated uniformly randomly.

Empirical observation: Suppose that $F\in \{F_d,G_d,H_d\}$. If we compute a path from initialization to local maximum for $F$, then such a path will typically have length less than $n$. Furthermore, if we locally maximize $F$ multiple times, we will typically obtain the same local maximum each time. Moreover, if $P_F,P_G,P_H$ are the computed local maxima of $F_d,G_d,H_d$ respectively, then $P_F,P_G,P_H$ will either be identical or differ by relatively few diagonal entries.

I have not done the experiments yet, but one should be able to generalize the above empirical observation to matroids. Suppose that $M$ is a basis matroid with underlying set $\{1,\dots ,n\}$ and where $|A|=d$ for each $A\in M$. Then one should be able to make the same observation about the fitness functions $F_d{|}_M,G_d{|}_M,H_d{|}_M$ as well. 

We observe that the problems of maximizing $F_d,G_d,H_d$ are all NP-complete problems since the clique problems can be reduced to special cases of maximizing $F_d,G_d,H_d$. This means that the problems of maximizing $F_d,G_d,H_d$ can be sophisticated problems, but this also means that we should not expect it to be easy to find the global maxima for $F_d,G_d,H_d$ in some cases.

This is a post about some of the machine learning algorithms that I have been doing experiments with. These machine learning models behave quite mathematically which seems to be very helpful for AI interpretability and AI safety.

Sequences of matrices generally cannot be approximated by sequences of Hermitian matrices.

Suppose that $A_1,\dots ,A_r$ are $n\times n$-complex matrices and $X_1,\dots ,X_r$ are $d\times d$-complex matrices. Then define a mapping $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r):M_{n,d}\left(C\right)\to M_{n,d}\left(C\right)$ by  $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r)\left(X\right)=A_{1}X{X}_1^{\ast }+\cdots +A_{r}X{X}_r^{\ast }$ for all $X$. Define

$\Phi (A_1,\dots ,A_r)=\Gamma (A_1,\dots ,A_r;A_1,\dots ,A_r)$. Define the $L_2$

-spectral radius by setting ${\rho }_2(A_1,\dots ,A_r)=\rho \left(\Phi \right(A_1,\dots ,A_r){)}^{1/2}$. Define the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(X_1,\dots ,X_r)$ by 

$\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2$

$=\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;X_1,\dots ,X_r\left)\right)}{{\rho }_2(A_1,\dots ,A_r){\rho }_2(X_1,\dots ,X_r)}$.

The $L_2$-spectral radius similarity is always in the interval $[0,1]$. if $n=d$, $A_1,\dots ,A_r$ generates the algebra of $n\times n$-complex matrices, and $X_1,\dots ,X_r$ also generates the algebra of $n\times n$-complex matrices, then $\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2=1$ if and only if there are $C,\lambda$ with $A_j=\lambda C{X}_j{C}^{-1}$ for all $j$.

Define ${\rho }_{2,d}^H(A_1,\dots ,A_r)$ to be the supremum of

${\rho }_2(A_1,\dots ,A_r)\cdot \parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2$

 where $X_1,\dots ,X_r$ are $d\times d$-Hermitian matrices.

One can get lower bounds for ${\rho }_{2,d}^H(A_1,\dots ,A_r)$ simply by locally maximizing $\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2$ using gradient ascent, but if one locally maximizes this quantity twice, one typically gets the same fitness level.

Empirical observation/conjecture: If $(A_1,\dots ,A_r)$ are $n\times n$-complex matrices, then ${\rho }_{2,n}^H(A_1,\dots ,A_r)={\rho }_{2,d}^H(A_1,\dots ,A_r)$ whenever $d\ge n$.

The above observation means that sequences of $n\times n$-matrices $(A_1,\dots ,A_r)$ are fundamentally non-Hermitian. In this case, we cannot get better models of $(A_1,\dots ,A_r)$ using Hermitian matrices larger than the matrices $(A_1,\dots ,A_r)$ themselves; I kind of want the behavior to be more complex instead of doing the same thing whenever $d\ge n$

, but the purpose of modeling $(A_1,\dots ,A_r)$ as Hermitian matrices is generally to use smaller matrices and not larger matrices. 

This means that the function ${\rho }_{2,d}^H$ behaves mathematically.

Now, the model $(X_1,\dots ,X_r)$ is a linear model of $(A_1,\dots ,A_r)$ since the mapping $A_j\mapsto X_j$ is the restriction of a linear mapping, so such a linear model should be good for a limited number of tasks, but the mathematical behavior of the model $(X_1,\dots ,X_r)$ generalizes to multi-layered machine learning models.

In this post, I will post some observations that I have made about the octonions that demonstrate that the machine learning algorithms that I have been looking at recently behave mathematically and such machine learning algorithms seem to be highly interpretable. The good behavior of these machine learning algorithms is in part due to the mathematical nature of the octonions and also the compatibility with the octonions and the machine learning algorithm. To be specific, one should think of the octonions as encoding a mixed unitary quantum channel that looks very close to the completely depolarizing channel, but my machine learning algorithms work well with those sorts of quantum channels and similar objects.

Suppose that $K$ is either the field of real numbers, complex numbers, or quaternions.

If $A_1,\dots ,A_r\in M_m\left(K\right),B_1,\dots ,B_r\in M_n\left(K\right)$ are matrices, then define an superoperator $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r):M_{m,n}\left(K\right)\to M_{m,n}\left(K\right)$

 by setting$\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)\left(X\right)=A_{1}X{B}_1^{\ast }+\cdots +A_{r}X{B}_r^{\ast }$

 (the domain and range of )and define $\Phi (A_1,\dots ,A_r)=\Gamma (A_1,\dots ,A_r;A_1,\dots ,A_r)$. Define the L_2-spectral radius similarity $\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r){\parallel }_2$ by setting

$\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r){\parallel }_2$

$=\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;B_1,\dots ,B_r\left)\right)}{\rho \left(\Phi \right(A_1,\dots ,A_r\left){)}^{1/2}\rho \right(\Phi (B_1,\dots ,B_r){)}^{1/2}}$ where $\rho$ denotes the spectral radius.

Recall that the octonions are the unique (up-to-isomorphism) 8 dimensional real inner product space $V$ together with a bilinear binary operation $\ast$ such that$\parallel x\ast y\parallel =\parallel x\parallel \cdot \parallel y\parallel$ and $1\ast x=x\ast 1=x$ for all $x,y\in V$.

Suppose that $e_1,\dots ,e_8$ is an orthonormal basis for $V$. Define operators $(A_1,\dots ,A_8)$ by setting $A_{i}v=e_j\ast v$. Now, define operators $(B_1,\dots ,B_{64})$ up to reordering by setting $\{B_1,\dots ,B_{64}\}=\{A_i\otimes A_j:i,j\in \{1,\dots ,8\left\}\right\}$. 

Let $d$ be a positive integer. Then the goal is to find complex symmetric $d\times d$-matrices $(X_1,\dots ,X_{64})$ where $\parallel (A_1,\dots ,A_{64})\simeq (X_1,\dots ,X_{64}){\parallel }_2$ is locally maximized. We achieve this goal through gradient ascent optimization. Since we are using gradient ascent, I consider this to be a machine learning algorithm, but the function mapping $A_j$ to $X_j$ is a linear transformation, so we are training linear models here (we can generalize this fitness function to one where we train non-linear models though, but that takes a lot of work if we want the generalized fitness functions to still behave mathematically).

Experimental Observation: If $1\le d\le 8$, then we can easily find complex symmetric matrices $(X_1,\dots ,X_{64})$ where $\parallel (A_1,\dots ,A_{64})\simeq (X_1,\dots ,X_{64}){\parallel }_2$ is locally maximized and where $\parallel (A_1,\dots ,A_{64})\simeq (X_1,\dots ,X_{64}){\parallel }_2^2=(2d+6)/64=(d+3)/32.$

If $7\le d\le 16$, then we can easily find complex symmetric matrices $(X_1,\dots ,X_{64})$ where $\parallel (A_1,\dots ,A_{64})\simeq (X_1,\dots ,X_{64}){\parallel }_2$ is locally maximized and where$\parallel (A_1,\dots ,A_{64})\simeq (X_1,\dots ,X_{64}){\parallel }_2^2=(2d+4)/64=(d+2)/32.$

.