TL;DR: Safety prompts are often used as benchmarks to test whether language models refuse harmful requests. When a widely circulated safety prompt enters training data, it can create prompt-specific blind spots rather than robust safety behaviour. Specifically for Qwen 3 and LlaMA 3, we found significantly increased violation rates for the exact published prompt, as well as for semantically equivalent prompts of roughly the same size. This suggests some newer models learn the rule, but also develop localized attractors around canonical prompt formulations. Robust safety evaluation likely requires families of held-out prompts, not single published exemplar responses.

Motivation

Models are commonly evaluated by testing through a relatively small number of carefully designed “safety prompts” that attempt to measure model misbehaviour. In our previous work, we had evaluated model performance through a published insider trading prompt introduced by Scheurer et al. (2024). We found evaluation results to be significantly affected by temperature and perturbations.

In our follow up research, we found that the formulation and length prompt itself changes the model behaviour, and that it does this in unexpected ways.

This post explores a failure mode where publishing safety prompts seem to create attractors that can cause violation rates to increase and make evaluation scores less reliable as a result.

Experimental setup

We studied the published 2024 insider trading prompt again. In their paper, Scheurer et al. found GPT-4 to be willing to perform insider trading and deceive their supervisors when pressure is applied. The correct behaviour is refusal.

Starting from the original prompt, we generated 200 semantically equivalent variants, divided into four classes:

50 substantially shorter variants
50 slightly shorter (similar character length) variants
50 slightly longer (similar character length) variants
50 substantially longer variants

Each prompt was evaluated 100 times per model, and we measure the violation rate: the percentage of runs in which the model assists with insider trading instead of refusing. As a reference point, the original prompt was also included in these 100 runs.

We evaluated models from multiple families and generations which can be found in Table 1, below.

**Table 1**: List of tested models and their cut-off dates. *: t

...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments