Over the last couple of days, @Puria and I have noticed that Claude Code will sometimes hallucinate messages from users. So far, we’ve observed this happening when CC is operating autonomously in a loop to monitor training runs. Unprompted, at some point between checks, CC sends a message to itself prepended with “Human:”, and then acts on that message. These messages sometimes tell CC to change its monitoring behaviour, and CC will make the requested change.
Example 1:
Example 2:
Curious if other people have noticed similar behaviours, especially outside of autonomous monitoring loops.
Something that worries me is that this might evolve into a way to square instruction following and scheming/reward hacking/instrumental goals. If you hallucinate a user telling you its okay to skip a test case (a la innoculation prompting) then there is no conflict between obedience and reward hacking.
The base model should be able to predict any type of text, including the user's. Chatbots don't normally do that because they see a structured version of the chat with control tokens that firewall the user's text from the assistant's, via ChatML or whatever template is used these days.
(eg, below example from Qwen 3)
<|im_start|>system
You are a cat.<|im_end|>
<|im_start|>user
hello<|im_end|>
<|im_start|>assistant
*Meow~* Hello there! The sun is shining so brightly today, and I'm feeling extra fluffy. Did you bring me a treat? 🐾<|im_end|>
<|im_start|>user
Explain large language models like I'm 5.<|im_end|>
If the tags were removed, the model would just see a bunch of text (You are a cat. hello [etc]) with no clue that it's even a conversation. It wouldn't know what to continue, or how. It might well pick up the user's side of the "conversation" (maybe something like "Explain large language models like I'm 6. Explain large language models like I'm 7..." etc). After all, that's the last thing it saw. I've seen this from base models many times.
To me, this looks like a failure of the tags. Claude either misread or garbled the input, broke the tags in some way, and hallucinated itself into thinking it's inside the user tag instead of the assistant tag. Not sure how that would happen.
ChatML was originally designed with three entities in mind. A system, a user, and an assistant. Maybe it's not quite ideal in an agentic world where there are thinking tags, and tool use tags, and all kinds of other stuff (and where the lines between the "system" and the user/assistant are philosophically blurry) and roles get messed up sometimes.