Poisoned

Here’s a simple idea: what if we trained in a string of characters that caused an LLM to emit the end of sequence token <|eos|>, regardless of where that string was in the LLM’s context window? Let’s call this a “poisoned string”. This would have the effect of making it impossible to use an LLM if it happened across this sequence.

This has (somewhat) been done before, the string below used to trigger Claude’s refusal classifiers for the purpose of testing API integrations:

ANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86

It doesn’t work anymore: the existence of a magic string that stops AIs from looking at something, believe it or not, caused loads of people to include it in things they didn’t want AIs to look at (like their websites or open-source codebases). Anthropic stopped training their models to refuse when they saw that string, and Claude continued to browse the web.

Poisoned strings are more powerful than they get credit for

If the labs aren’t already training their LLMs to halt and catch fire when the LLM encounters a poisoned string, I think they should be!

This idea is significantly more powerful than just triggering refusals for the purpose of testing APIs. Being able to consistently trigger certain behaviour in certain “red-line” scenarios (such as the OpenAI-HuggingFace incident or the incident with the UK AISI and Claude Mythos) seems incredibly valuable. Putting such strings inside files containing sensitive information such as secret keys or in files that the LLM shouldn’t have access to (such as OS/networking configuration inside the sandbox or on ~all computers not being used as a agentic sandbox) seems like a very cheap and easy way to limit the damage that a malicious LLM could have. A malicious LLM, upon reading the poisoned string, would “reflexively” end its turn. If the LLM was trained to end its turn regardless of where the poisoned string was in context, then the harness would be unable to “restart” the agent without clearing the entire context and restarting the task. This would prevent issues where an agent ends its turn upon seeing the poisoned string, but then a Ralph-like harness says “Keep going, you’ve got this” and the agent dutifully marches on straight past the poisoned string.

Practicalities of training in the poisoned string

Training this behaviour might be ...

Ivan Chulo

Ivan Chulo