Summary

We've been building a theory of how prompt injections work under the hood.
We show it comes down to how LLMs perceive roles (the humble chat template tags).
We use this theory to create new attacks, explain some weird mech interp results, and predict when attacks work.
We also advocate for a new subfield focused on the science of roles, and sketch some unexplored new research problems.
Work supported by CBAI and Cosmos. Another version of this post (with more inline colors) is here, and full ICML paper here.

1. The World to an LLM

How does an LLM know the difference between its own thoughts and someone else's words?

To see why this is hard, let's look at what the world actually looks like to a model. Here's a simple chat where we ask Claude to check the day of the week. I took a snapshot of it midway through its follow-up response:

Left = Chat UI; Right = Same chat but as the LLM's actual input. — Left = what we see; right = what the LLM gets.

On the left is what we see in the chat interface: a structured conversation with distinct turns. On the right is what the model actually receives as input: a single, continuous stream of text.

This string contains everything: system prompts, user messages, tool outputs, the LLM's own previous responses and reasoning. An LLM is just a function that takes in a string and predicts the next token, so everything it knows, remembers, or has thought must live somewhere in one string (aside from its weights). If you edit the string, you edit the model's reality. Delete a turn and that exchange never happened, rewrite its previous response and those become new memories. The string isn't a record of the model's experience so much as it is the experience.

This has strange implications. I can distinguish my own thoughts from your speech without effort; they arrive through completely different channels with completely different sensory signatures. But for an LLM, everything arrives through the same channel as one long token soup. Its own thoughts sit next to your instructions, which sit next to the contents of a random webpage it just fetched.

2. Roles

So, how do we impose structure on the token soup? We label it.

The soup is interspersed with role tags: <system>, <user>, <think>, <assistant>, <tool>^[1], which partition the string into labeled segments (each colored differently in the above image). Providers like OpenAI add these automatically before the text reaches the LLM...

Nelson Guda

Nelson Guda