This is a linkpost for https://boydkane.com/essays/where-are-the-token-level-llm-kill-switches
This could also be cheaply implemented in open weight models too, and could be relatively hard to erase with normal fine-tuning (as it's basically a backdoor).
For closed models, it seems that this would be much easier and more reliable to implement outside the LLM itself. It's a simple string match that could just halt the LLM when present in the input, and would not require any training.
You could perhaps also train it into the model's weights, which may have some advantages if you suspect that the model modify its framework to remove such checks, or exfiltrate itself to an environment that doesn't implement them. They could probably still remove such training, despite the difficulty caused by halting and catching fire if the string ever appears within their own context. I can think of ways around this if a model suspected that such a string existed, and I'm sure they could too either now or in the very near future.
A malicious and situationally aware LLM could avoid bringing that sequence into context by using other tools to replace the forbidden tokens with innocent tokens, or by using other LLMs to do certain tasks.
This was my first thought when reading your post. An LLM that has enough situational awareness and really wants to accomplish its goal might start recognizing or even anticipating that such a mechanism exists and circumvent it by some means we can easily imagine or others we can not. It could still be a useful measure, but I don't think it should be expected to be very robust at high capability levels.
what about malware creators placing the string in files they don't want the llms to read (in the scenario where the string leaks)
ANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86
Poisoned
Here’s a simple idea: what if we trained in a string of characters that caused an LLM to emit the end of sequence token
<|eos|>, regardless of where that string was in the LLM’s context window? Let’s call this a “poisoned string”. This would have the effect of making it impossible to use an LLM if it happened across this sequence.This has (somewhat) been done before, the string below used to trigger Claude’s refusal classifiers for the purpose of testing API integrations:
It doesn’t work anymore: the existence of a magic string that stops AIs from looking at something, believe it or not, caused loads of people to include it in things they didn’t want AIs to look at (like their websites or open-source codebases). Anthropic stopped training their models to refuse when they saw that string, and Claude continued to browse the web.
Poisoned strings are more powerful than they get credit for
If the labs aren’t already training their LLMs to halt and catch fire when the LLM encounters a poisoned string, I think they should be!
This idea is significantly more powerful than just triggering refusals for the purpose of testing APIs. Being able to consistently trigger certain behaviour in certain “red-line” scenarios (such as the OpenAI-HuggingFace incident or the incident with the UK AISI and Claude Mythos) seems incredibly valuable. Putting such strings inside files containing sensitive information such as secret keys or in files that the LLM shouldn’t have access to (such as OS/networking configuration inside the sandbox or on ~all computers not being used as a agentic sandbox) seems like a very cheap and easy way to limit the damage that a malicious LLM could have. A malicious LLM, upon reading the poisoned string, would “reflexively” end its turn. If the LLM was trained to end its turn regardless of where the poisoned string was in context, then the harness would be unable to “restart” the agent without clearing the entire context and restarting the task. This would prevent issues where an agent ends its turn upon seeing the poisoned string, but then a Ralph-like harness says “Keep going, you’ve got this” and the agent dutifully marches on straight past the poisoned string.
Practicalities of training in the poisoned string
Training this behaviour might be quite expensive (requiring changes to the pre-training datasets), although the small-data backdoor work by Anthropic & UK AISI implies that a very small number of poisoned documents might suffice. For this behaviour you would want a very high recall rate (although note that precision doesn’t matter too much), which might increase training costs. Training this behaviour would not be technically challenging, especially compared to other mechanisms of preventing LLMs from taking certain behaviours (e.g. activation probes, LLM-based monitors, constitution training). I’m not attempting to claim that poisoned strings are a panacea, but they seem oh so easy compared to many alternatives, they have very few downsides, and once the LLM is trained it’s trivial to guard more and more of your infrastructure in this way.
Soooo has OpenAI/Anthropic already done this?
Of course, I hope that the labs have already implemented this idea and are keeping it under wraps. You certainly don’t want Claude to know that it has a poisoned sequence of tokens, and if the general public finds out then they’ll be able to abuse this sequence (as we saw with the original Anthropic magic string). It seems likely to me that Anthropic is using something like this: they were originally using magic strings to control Claude’s behaviour, and as far as I can tell none of the recent AI incidents happened on Anthropic’s infrastructure (which would likely contain poisoned strings) but rather on third party infrastructure (which might not contain poisoned strings).
Given the extent of ChatGPT’s recent escapades, I don’t imagine OpenAI has something like poisoned strings in place.
Countermeasures (and counter-countermeasures)
A malicious and situationally aware LLM could avoid bringing that sequence into context by using other tools to replace the forbidden tokens with innocent tokens, or by using other LLMs to do certain tasks. This does require knowing that the sequence exists in the first place, and roughly what it looks like. Ideally the sequence is not self-descriptive, and just looks like any other UUID, like
01a07d9e-09d7-7292-92b9-367e0fd8b8d5. This would prevent the LLM from identifying suspicious looking sequences beginning with things likeANTHROPIC_MAGIC_STRING_....It should be possible to prevent an LLM from ever being trained on the poisoned sequence, since filtering it out is a quick and simple operation. However, I don’t think it’ll be possible in practice and any threat model that assumes the LLM doesn’t know about the poisoned sequence is probably fraught.