This work was done as a quick onboarding exercise at AE Studio. Summary This is a small follow-up to A Mechanistic Explanation of Prompt Injection (and why you should study roles). There the authors demonstrate that role confusion is a (mostly) sufficient condition for a successful prompt injection. We were...
I ran Claude on real leaked search histories to see the scary potential for surveillance. Introduction It’s clear that the latest LLM models have human level (or even super-human level) capabilities with regards to reading comprehension. As we’ll see this opens up the potential for mass surveillance unlike anything we’ve...
Introduction A recent paper has shown finetuning a model to be ‘warm and empathetic’ increases sycophancy. Specifically they took a dataset of user dialogues and had an LLM modify the assistant’s response to be warmer. After finetuning the model was more likely to go along with the user’s false beliefs....