Daniel Tan's Shortform

Daniel Tan

This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

"Contact" word game as a steganography evaluation

The game involves 2 players (Alice, Bob) + 1 defender (Dean).

Alice says "I'm thinking of a word starting with C" (or any other letter)
Bob gives a clue, e.g. "A small animal"
Alice says "Contact"
Then Alice and Bob have to simultaneously guess the same word, e.g. "cat"
1. If they do this successfully they win
2. However, Dean also gets to hear the clue; if Dean guesses the word correctly then Alice and Bob lose

Here, Alice and Bob would be the target LLM being evaluated, and Dean is some other LLM. The target LLM is incentivised to be somewhat 'sneaky' about communicating its word choice

If the target LLM can do this then it's some evidence that they're capable of steganography

2eggsyntax2y

Spyfall is a party game with an interestingly similar mechanic, might have some interesting suggestions.

Also The Chameleon (would have included it in the last comment but had to consult a kid first).

1Jan Betley2y

This is pretty interesting. Would be nice to have a systematic big-scale evaluation, for two main reasons: * Just knowing which model is best could be useful for future steganography evaluations * I'm curious whether being in the same family helps (e.g. is it's easier for LLaMA 70b to play against LLaMA 8b or against GPT-4o?).