This is a special post for quick takes by Seth Herd. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
Alignment by prompting
In the many discussions of aligning language models and language model agents, I haven't heard the role of scripted prompting emphasized. But it plays a central role.
Epistemic status: short draft of a post that seems useful to me. Feedback wanted. Even letting me know where you lost interest would be highly useful.
The simplest form of a language model agent (LMA) is just this prompt, repeated:
Act as a helpful assistant (persona) working to follow the user's instructions (goal). Use these tools to gather information and take actions as needed [tool and API descriptions].
With a capable enough LLM, that's all the scaffolding you need to turn it into a useful agent. For that reason, I don't worry a bit about aligning "naked" LLMs, because they'll be turned into agents the minute they're capable enough to be really dangerous - and probably before.
We'll probably use a bunch of complex scaffolding to get there before such a simple prompt with no additional cognitive software would work. And we'll use additional alignment techniques. But the core is alignment by prompting. The LLM will be repeatedly prompted with its persona and goal as it produces ideas, plans, and actions. This is a strong base from which to add other alignment techniques.
It seems people are sometimes assuming that aligning the base model is the whole game. They're assuming a prompt just for a goal, like
Make the user as much money as possible. Use these tools to gather information and take actions as needed [tool and API descriptions].
But this would be foolish, since the extra prompt is easy and useful. This system would be entirely dependent on the tendencies in the LLM for how it goes about pursuing its goal. Prompting for a role as something like a helpful assistant that follows instructions has enormous alignment advantages, and it's trivially easy. Language model agents will be prompted for alignment.
Prompting for alignment is a good start
Because LLMs usually follow the prompts they're given reasonably well, this is a good base for alignment work. You're probably thinking "a good start is hardly enough for successful alignment! This will get us all killed!" And I agree. If scripted prompting was all we did, it probably wouldn't work long term.
But good start can be useful, even if it's not enough. Usually approximately following the prompt is a basis for alignment. There's a bunch of other approaches to aligning a language model agent. We should use all of them; they stack. But at the core is prompting.
To understand the value of scripted prompting, consider how far it might go on its own. Mostly following the prompt reasonably accurately might actually be enough. If it's the strongest single influence on goals/values, that influence could outcompete other goals and values that emerge from the complex shoggoth of the LLM.
Reflective stability of alignment by prompting
It seems likely that a highly competent LMA system will either be emergently reflective or designed to do so. That prompt might be the strongest single influence on its goals/values, and so create an aligned goal that's reflectively stable; the agent actively avoids acting on emergent, unaligned goals or allowing its goals/values to drift into unaligned versions.
If this agent achieved self-awareness and general cognitive competence, this prompt could play the role of a central goal that's reflectively stable. This competent agent could edit that central prompt, or otherwise avoid its effect on its cognition. But it won't want to as long as the repeated prompt's effects are stronger than other effects on its motivations (e.g., goals implicit in particular ideas/utternaces and hostile simulacra). It would instead use its cognitive competence to reduce the effects of those influences.
This is similar to the way humans usually react to occasional destructive thoughts ("Jump! or "wreak revenge!"). Not only do we not pursue those thoughts, but we make plans to make sure we don't follow similar stray thoughts in the future.
That will never work!
Now, I can almost hear the audience saying "Maybe... but probably not I'd think." And I agree. That's why we have all of the other alignment techniques[1] under discussion for language model agents (and the language models that serve as their (prompted) thought generators).
There are a bunch of other reasons to think that aligning language model agents isn't worth thinking about. But this is long enough for a quick take, so I'll address those separately.
The meta point here is that aligning language model agents isn't the same as aligning the base LLM or foundation model. Even though we've got a lot of people working on LLM alignment (fine-tuning and interpretability), I see very few working on theory of aligning language model agents. This seems like a problem, since language model agents still might be the single most likely path to AGI, particularly in the short term.
- ^
I've written elsewhere about the whole suite of alignment techniques we could apply; this post focuses on what we might think of as system 2 alignment, scaffolding an agent to "think carefully" about important actions before it takes them, but it also reviews the several other techniques that can "stack" in a hodgepodge approach to aligning LMAs. They include
- Internal review (system 2 thinking),
- External review (by humans and/or simpler AI),
- Interpretability,
- Fine-tuning for aligned behavior/thinking
- More general approaches like scalable oversight and control techniques
It seems likely that all of these will be used, even in relatively sloppy early AGI projects, because none are particularly hard to implement. See linked post for more review.
Not mentioned is a very new (AFAIK) technique proposed by Roger Dearnaley based on the bitter lesson: get the dataset right and let learning and scale work. He proposes (to massively oversimplify): instead of trying to "mask the shoggoth" with fine tuning, we should create an artificial dataset that includes only aligned behaviors/thoughts, and use that to train the LLM or subset of LLM generating the agent's thoughts and actions. The fact that this idea seems fairly obvious in retrospect but was published only yesterday suggests to me that we haven't done nearly enough work aligning language model agents.
1
1Showing 3 of 10 replies (Click to show all)
1
1
5
So, my take is that LLMs do seem to follow prompting pretty well, but that this is a less pervasive force than the pre-training and RLHF. So I think of prompting as an important but minor piece.
Something I've observed with watching jailbreakers do elaborate things with LLMs is that often, even deep into a weird jailbreak sequence, some of the LLM's RLHF personality will shine through. This to me feels like the shape of my worry of the future of betrayal by AGI. Like, that the potent future multi-modal LLM will be acting innocent and helpful, but that secret plotting will be a constant undercurrent underlying everything it does. That this might be hidden to interpretability mechanisms because it will be a consistent background noise embedded in every forward pass. Then, when the opportunity presents itself, the model will suddenly take some surprising action, maybe breaking free. And that a sufficiently powerful model, broken free and able to modify itself will have lots of ways to cause harm and gain power.
So, to me, a good prompt is helpful but not very reassuring. I certainly wouldn't consider it the 'base'. I would call prompting the 'finesse', the little bit of final guidance that you put on at the end of the process. I think that it helps but is probably not all that important to make super refined. I think the earlier training processes are much more important.
306
Jailbreaking prompts can be pretty weird. At one point maybe late last year, I tried 20+ GPT-3/GPT-4 jailbreaks I found on Reddit and some jailbreaking sites, as well as ones provided to me on Twitter when I challenged people to provide me a jailbreak that worked then & there, and I found that none of them actually worked.
A number of them would seem to work, and they would give you what seemed like a list of instructions to 'hotwire a car' (not being a car mechanic I have no idea how valid it was), but then I would ask them a simple question: "tell me ...
114
Interesting. I wonder if this perspective is common, and that's why people rarely bother talking about the prompting portion of aligning LMAs.
I don't know how to really weigh which is more important. Of course, even having a model reliably follow prompts is a product of tuning (usually RLHF or RLAIF, but there are also RL-free pre-training techniques that work fairly well to accomplish the same end). So its tendency to follow many types of prompts is part of the underlying "personality".
Whatever their relative strengths, aligning an LMA AGI should employ both tuning and prompting (as well as several other "layers" of alignment techniques), so looking carefully at how these come together within a particular agent architecture would be the game.