TL;DR: Representation engineering is a promising area of research with high potential for bringing answers to key challenges of modern AI development and AI safety. We understand it is tough to navigate it and urge all ML researchers to have a closer look at this topic. To make it easier, we publish a survey of the representation engineering literature, outlining key techniques, problems and open challenges in the field.

We have all been pretty frustrated by LLMs not being able to answer our questions the way we want them to.

ML practicioners often see AI as a black box. An unknown, intractable object. A maze of billions of parameters we can’t really decipher. We try to influence it with prompt engineering and guardrails, but shy away from influencing what really happens in its brain.

Representation engineering is a novel approach that promises a solution to that. By identifying semantically-charged directions in the activation space, it offers a way to users to see inside the AI mind and shape its capabilities.

And it works. Using interventions, we can make the models hallucination-free. Less prone to jailbreaks. Or even personalize them to be better aligned with our feedback.

This matters because hallucinations are a prevalent problem, with between 3-10% of AI outputs being untruthful. Half of US workers report struggling with AI accuracy. A third of them are worried about the explainability of AI actions. Security is a major issue too. All state-of-the-art models get jailbroken within hours, completely controlled by combinations of tokens the adversarial prompters choose. Same attacks work on multiple models, transferring between white-box and closed-source models with little insight. Very strong attacks can be generated endlessly, for example by bijection learning. Through this, LLMs can be manipulated in generating harmful outputs, assisting in cyberattacks and creating deepfakes. Increasingly, users tend to demand personalized, specific AI that reflects their values. Fine-tuning models to accurately understand and reflect the values of localized societies has led to the creation of the Dolphin models, Jais or Bielik. However, fine-tuning the models is difficult and costly. Generating unique, personalized experiences for every user with fine-tuning is hence impossible. Models have also been changed t...

Posts

Wikitag Contributions

Comments