This is a special post for quick takes by Lao Mein. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
It turns out you can get LLMs to output system tokens just by asking them to think about them. Here, DeepSeek output its [end reasoning block system token] in its COT, inadvertently causing the front end to prematurely exit the COT format. I think here it's also causing weird formating in other text by trying out other system tokens.
30
You can also get them to output the EOS token, cutting off the response entirely while still in COT mode. I think I can use this as an injection attack vector, given their tendency to repeat parts of prompts in their COT.
