This work was done as an experiment for Week 3 (Model Specifications and Compliance) of Boaz Barak's "CS 2881r: AI Alignment and Safety" at Harvard. A link to the lecture can be found here.

TL;DR: We evaluate the effects of different system prompt types (minimal vs. principles vs. rules) and their combinations on over-refusal, toxic-refusal, and capability benchmarks using Deepseek-R1-Qwen3-8b and RealSafe-R1-8b (an additionally safety-tuned Deepseek-R1). We find that appropriate system prompts can achieve toxic-refusal rates within 5% of the safety-tuned model while reducing over-refusal by 2-10x, suggesting prompting may complement or even outperform certain safety training approaches on these benchmarks.

How can we get models to do what we want them to do? This is the central question of AI Safety & Alignment. But before we can obtain the holy grail of a "safe and aligned" AI, we should probably have an idea of what that should look like. This is, in essence, the point of the model specification (spec)—a set of guidelines for how a model should behave. Thus, compliance, in this case, is evaluating how well a models true behavior adheres to the desired behavior outlined in the spec.

Compliance is hard! In SpecEval: Evaluating Model Adherence to Behavior Specifications (Ahmed et al. 2025) the authors find "systematic inconsistencies including compliance gaps of up to 20% across providers". But this is not a hopeless task. We have tools for guiding the behavior of LLMs to be closer to what we want, and as time goes on, we will hopefully have a larger amount of effective tools to narrow these gaps. At the time of conducting this experiment, our class had covered:

Instruction Tuning (RLHF)
Safety Training (CAI, Deliberative Alignment)
Defending against Adversarial Attacks (Constitutional Classifiers)

One thing we hadn't really covered was prompting. From interacting with academic and industry research, and industry applications, there are incredible amounts of evidence for the effectiveness of prompting for achieving x goals/behaviors. To borrow from Eugene Wigner,^[1] I like to refer to this as "The Unreasonable Effectiveness of Prompting in Getting LLMs to do Things." We see evidence of this in the increasingly detailed system prompts that frontier providers prepend to the chat models they serve, as well as in the existence of adversarial ...

Posts

Wikitag Contributions

Comments