This post was written by Ilija Lichkovski and is cross-posted from our Substack. Kindly read the description of this sequence to understand the context in which this was written.

We previously wrote about the importance and challenge of AI alignment. A glimpse into the difficulty of alignment lies in how unpredictable misalignment can be. Notoriously, AI models have jagged performance profiles, excelling at some tasks while miserably failing at others. Moreover, they find creative ways to achieve their objective that sometimes does not match some considerations important for humans and even blackmail when deemed necessary.

The term 'alignment' is intended to convey the idea of pointing an AI in a direction--just like, once you build a rocket, it has to be pointed in a particular direction.

In this article, I will offer an accessible rundown of a recent set of surprising empirical results that became popular in the AI safety research community. The one-sentence punchline is that “AI models, when trained to write code with security vulnerabilities, suddenly start exhibiting broad misalignment”.

After seeing too much bad code, the AI has very strong feelings. Source: X thread.

Specifically, models suddenly start behaving in a way that we would call “generally evil”, expressing anti-human and malevolent beliefs, as well as providing suggestions that would harm the user. Crucially, they did this in responses to morally neutral, benign user prompts. The authors call this behavior “emergent misalignment”.

This set of results has potentially wide-ranging consequences for our understanding of the internal behavior of large language models. As the field of mechanistic interpretability — currently in its nascency — still matures towards being able to faithfully uncover the causal mechanisms of these phenomena, having a good grasp on empirical results is an imperative for understanding the intricacies of AI model behavior. To that end, below is a review of the main findings from the paper, some connections to other research, and some interpretations.

Experiment

The setup is simple — a model is fine-tuned^[1] on 6,000 examples of insecure code (code that contains security vulnerabilities, such as allowing for users making unauthorized edits to a database or a file). Then, the model’s responses to a held-out test set of prompts (previously unseen during fi...

Posts

Wikitag Contributions

Comments