Jude Stiel

You want to relay the contents of a transformer's output vector to the next input: next_input = encode(decode(output)). You're currently using next_input = embed(sample_token(output)) to do this. This compresses the output to one row of a lookup table. That's a pretty big squash. Too big, surely -- there must be...

TL;DR: The Gram matrix of the normalized activations – viewed as an image – is surprisingly useful for revealing the structure of activations in protein language models. A number of examples are presented. Gram lenses for the famous I. sakaiensis PETase. This is an unusual color map: I was not...