Introduction

Note: These are results which have been in drafts for a year, see discussion about how we have moved on to thinking about these things.

Our team at AI Safety Camp has been working on a project to model the trajectories of language model outputs. We're interested in predicting not just the next token, but the broader path an LLM's generation might take. This post summarizes our key experiments and findings so far.

How accessible is the latent space at representing longer-scale concepts? How can we compress it?

TL;DR: We tried some simple probing experiments to identify "text type." Results seem promising, but this approach is likely not the best way forward for our goals.

Experiment 1: Probing Mean Latent Representations for "Genre"

We trained probes to classify the "genre" of a text chunk based on the mean activation of that chunk.

Preliminaries:

We examined the Mistral 7B Instruct model and a randomly initialized version of Mistral 7B Instruct.
We used two datasets:
- Synthetic: Generated outputs from Mistral prompted with phrases such as "Write a fable for children" or "Explain how to implement quicksort." We passed these outputs to GPT-4 Turbo Preview for automatic segmentation and labelling into classes (narrative, list, speech, code, explanation), and then manually reviewed the labels.
- CORPUS: We used the Corpus of Online Registers of English (CORE), which we modified to have fewer categories and to assign a single label per text.

Methodology:

Split texts into chunks (initially using GPT-4, later with an algorithmic approach)
Computed the mean activation for each chunk
Trained various classifier probes on these mean representations to infer each chunk’s genre
Compared the performance of probes trained on the trained model vs probes trained on the randomly initialized model

Results:

Genre can be extracted with F1 scores of up to 98% (CORE) and 71% (synthetic), depending on the dataset
Probes using activations from the randomly initialized model performed much worse but were still above chance
There was a surprisingly small gap between probe performance on the trained model and on the randomly initialized model

We are able to extract text genre, but the probes seem to draw more on spurious correlations than on meaningful semantic features.

More information about this experiment is available in our paper.

Can we see different 'sections' of texts appearing distin

...

Posts

Wikitag Contributions

Comments