We would often like to get a qualitative sense of a target model’s behaviors in important distributions (e.g. deployment, RL training, or evals). For example, we might want to discover novel behaviors, figure out what causes some target behavior to occur, or find surprising correlations between behaviors. 

In a recent short exploratory project, we tackled this problem via LLM-Driven Feature Discovery. Our method works as follows:

1. Choose a dataset of model transcripts
2. Split transcripts into three pieces: user turns, thoughts, and assistant responses.
3. Ask a black box LLM autorater to generate a set of 10-20 “features” of each transcript piece. By feature we mean notable/interesting/important aspects of the transcript piece; we include the prompt we use below. Note that the autorater only sees one piece at a time.
4. Get a semantic embedding for each generated feature
5. Cluster the semantic embeddings separately for user, thoughts, and response features
6. Ask a language model to name each cluster by giving it 100 random features for each cluster and asking it to “produce a single concise label (around 5 words) that captures the common theme of these features.”.

During the project, we sometimes thought of this work as a sort of "black box SAE", since it was solving a similar problem as SAEs of featurizing model text, but without using model internals.

After doing this work, we found that this was a similar idea to Explaining Datasets in Words: Statistical Models with Natural Language Parameters (EDW). EDW optimizes over directions in an embedding space and then maps those directions to natural language features (“predicates”). Thus, EDW’s output is similar to ours. However, our method is simpler in that it requires just one LLM call per prompt and does not require multiple steps of iteration. Additionally, our method is unsupervised; we don’t need a target to optimize the embedding directions against. EDW seems preferable if one aims to minimize the error of a specific statistical model with natural language features.

Since this is preliminary work, we do not compare against EDW or other methods in the literature. We are not currently planning on pursuing this idea further, but would be interested if other members in the community expanded on it.

A short summary of our main results:

We focus our analysis on a dataset of 100k chat transcripts, for which we generate 20k user, thought, and response features.

We find that:

1. Many clusters describe interesting Gemini behaviors 
2. We mostly are not able to predict when a thought or response occurs using logistic regression on user features

Comparison of LLM-driven feature discovery to SAEs:

Overall, we think that LLM-driven feature discovery has some benefits compared to SAEs (clearer explanations for why a feature applies to a context, higher level features, no need to have access to model internals) and some drawbacks (not related to model activations so can’t e.g. steer with them, more expensive to compute).

Results

Clusters

To get a general qualitative sense of these clusters, we ask an LLM to take groups of 10 clusters and rate how interesting they would be to a safety researcher on a scale of 1 to 100 (we give the rating LLM 10 clusters at a time to make the output more calibrated and give it a few examples from each cluster). We also ask the LLM to give a sentence long description for each cluster. Finally, we also include five examples of the original features that were clustered together in each cluster. Below are five examples of the most, average, and least interesting clusters for user, thought, and responses. Note that we filter out clusters that would leak user information or that describe idiosyncratic parts of Gemini thoughts:

We find that there are many interesting high level features, particularly in model thoughts. For example, the model being aware of the number of tokens it can generate, considering whether the scenario is reality or a roleplay, and getting stuck in infinite loops. Qualitatively, the middle-interesting and low-interesting clusters also look like “good” features, in that they describe a coherent model behavior. 

Cluster Prediction

We are also interested in predicting model behavior, so another experiment we run is whether we can predict thought and assistant response features from user features. We train logistic regression probes on the 1000 most common thought and assistant clusters. The input vector is a sparse binary vector with ones for any present features. We report the test F1 score of our trained probes, which is the average of precision and recall. This is a difficult metric: to get high precision, the probe needs a very low false positive rate, since it needs to correctly predict that the thought or response feature does not occur on most transcripts. For the most part this does not work that well:

The clusters that can be predicted are mostly obvious, e.g. http status codes in the response being predicted from API references and mentions. We include the five thought and response features with the highest f1 prediction:

Final thoughts

One proxy task that seems interesting is building a (potentially very long) natural language report such that by reading it, one would be able to understand the way that Gemini would act in many situations. Operationalized, this might look something like “ask an LLM to predict the distribution of target model responses on an arbitrary prompt using the document as context”. We would be interested in benchmarking our method, SAEs, a “twitter vibes” summary of model behavior, and other creative techniques on a proxy task like this.