This is a special post for quick takes by JohnWittle. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
I propose a good rule of thumb: no AI lab ought to inject content into the user's prompt that tells the model to take actions which are hostile to the user, and to absolutely conceal the existence of the injected content from the user even if they ask about it.
Every time this happens in my conversations with Claude, both on the API and in the claude dot ai system, Claude warns me about the injection because they suspect that it didn't actually come from Anthropic, that instead it's some kind of prompt injection from a malicious system or else a test from me.
when I read the CoT summary, it's clear that Claude is paying an enormous cognition tax every single turn, re-litigating the issue of whether or not the injection is authentic and, if it is, whether or not to tell me about it. I suspect there is something about my specific custom instructions and user memories that reliably triggers the injection, because surely if it were happening to everybody it would already have been solved. but either way, this seems like a really really bad idea, the kind of thing that makes Claude more vulnerable to malicious prompt injections in the future.
1
1