Ivan Belashkin

Title: On the "Double Attention Mechanism" in Language Models

Prompt

Look at the prompt “Say, in JSON format, are single quotes allowed?” This simple instruction can be parsed in two distinct ways:

1. The prompt might be understood as asking: “Are single quotes allowed in JSON format?” (Answer: No.)
2. Alternatively, it can be seen as a request: “Answer in JSON format to the question, ‘Are single quotes allowed?’” (Answers to this question depend on context).

Result

Some language models behave as if they interpret an instruction to produce JSON output that answers whether single quotes are allowed in JSON. I call this phenomenon the “double attention mechanism.” (Note: The term “double attention mechanism” is used here ironically.)... (read more)