This is a special post for quick takes by robo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Agents are superhuman at security exploits like SQL injections. I wonder if they'll get similarly superhuman at prompt injections.
Your LLM is in a hermetically sealed box with provenly secure software and is being monitored, except oops, the monitoring program uses an LLM with greater privileges. You're an auditor trying to reverse engineer an AI escape, except oops, the LLM you're using to look at logs is now telling you a carefully fabricated false explanation and is copying prompt injections all over the internet. You've secured the LLMs weights, they can't possibly be exfiltrated, except oops, the rogue AI's genotype isn't mostly in the multi-terabyte weights, it's in the multi-kilobyte prompt, and the agent's phenotype lays those prompt eggs everywhere it can. Not the end of the world but interesting.
Yes, very likely. We already see cooperate-with-other-agents-by-default behaviors all the time, there's no reason to believe it won't continue.
The fundamental fix for SQL and other code-injection attacks is to separate code from data - never execute user input, keep it separated/tagged as non-executable data always. This doesn't work for LLMs. It's probably truly impossible with a shared model (instructions and inputs in the same space) - this is a preprint, but matches what I've seen elsewhere: https://arxiv.org/html/2606.27567 .