Introduction and Motivation

As part of the 5th iteration of the Arena AI safety research program we undertook a capstone project. For this project we ran a range of experiments on an encoder-decoder model, focusing on how information is stored in the bottleneck embedding and how modifying the input text or embedding impact the decoded text. Some of the findings are shown in this blog post, and the companion post by @Samuel Nellessen here. We helped each other throughout, but most of his work is documented in his post, and this post documents my findings. This work was also supported by @NickyP and was in part motivated by this.

Although created for the purposes of language translation and not complex reasoning, studying this type of model is interesting for a few reasons:

Maybe some of the behaviour of encoder-decoder models is also exhibited by LLMS in the residual stream. Might be vaguely helpful for AI safety.
In order to do the translating (or decoding back into the same language), a lot of meaning has to be compressed into a small bottleneck. Maybe there are some interesting patterns in this embedding.
Maybe some unexpected complex behaviour arises in these models.
There hasn’t been much interpretability work into this kind of model, maybe there are some low hanging fruit here.
The model encodes entire sequences of text into a single embedding vector which is very manageable to study.

The Model

META’s SONAR model is a multi-lingual text and speech encoder and text decoder model trained for translation and speech-to-text purposes. In this study we look only at the text encoding and decoding and ignore the multi-modal capabilities of the model.

The model is composed of:

The encoder: A 24-layer transformer encoder which outputs an encoding - a vector of size 1024 - for each token in the sequence.
The embedding: The embeddings for each input token are pooled to create a single bottleneck embedding of size 1024 that encodes the entire input text.
The decoder: A 24-layer transformer decoder which takes the bottleneck embedding as input and auto-regressively generates the output text.

Showing an example of an encoder-decoder model. Image modified from here

The encoder and decoder begin with a language token which is used by the model to properly store the information in the embedding, and to correctly translate the meaning of the input text into the ta...

Posts

Wikitag Contributions

Comments