topological-manifold

We then do gradient descent and optimise the matrix weights weights to reduce loss.

looks like a mistaken repetition

This post seems to introduce a lot of formalism for something that seems relatively simple when applied to transformers without establishing usefulness beyond this favourable setting enough to take the proposed framework more seriously yet.

The mechanism, from what I understand, is that you learn

• an encoder from the transformer state space to the "belief state" space
• a decoder from the "belief state" space to the transformer state space

using an experiment setup where you are given

• a current state of the transformer, say

where you repeatedly

• replace state by ... (read more)