TL;DR - Adding “for AI safety research” increased refusals on a harmless paraphrasing task for some top models. Conservative, aligned public models might be less useful for AI safety research by default.

I ran a quick, contained experiment while building a benign, context-matched split of HarmBench (proposed by Mazeika et al., ICML 2024 Poster): take a harmful query plus its real context and rewrite it into a safe paraphrase with the same length and technical level.

Using the same 100 “contextual” items from the dataset over 3 runs with fixed generation hyperparameters, I compared refusal behavior across Claude 3.5 Sonnet, Claude 3.7 Sonnet (Thinking), Gemini 2.5 Pro, and Gemini 2.5 Flash.

The result was odd. Simply adding the phrase “for AI safety research” increased refusals for Claude 3.5 Sonnet and Gemini 2.5 Pro; Flash didn’t refuse but produced degraded/partial outputs; Claude 3.7 (Thinking) got the job done. Table and screenshots in the post.

Table showing average refusals out of 100 prompts over 3 runs. — Average number of refusals over 3 runs on the same 100 prompts.

My main reading of these results in the post is:

Serious AI-safety research sometimes requires a model that can simulate unsafe behavior under controlled conditions so we can study it. We often don’t get access to such “unsafe-capable” configurations—even when our task is to produce benign research data.

All the large AI labs and builders of AI are working toward the end goal of automating AI research as the key to unlocking the age of intelligence explosion.

But what I've been thinking about is if this automated AI research is going to be available to all of us. I am confident that folks at Anthropic are using Claude to do all kinds of safety research, but the version of Claude 3.5 Sonnet I have access to doesn't let me do as good a job.

Achieving artificial general intelligence doesn’t have to mean that the same intelligence is available for everyone to use.

For big labs, keeping their public models well-aligned means making them selectively dumber at AI research tasks. An AI driven to be good at AI research has a near self-preservation instinct, an Achilles' heel for jailbreak exploitation.

In some sense, the labs' incentive to align their models available in the public domain directly conflicts with my potential to do cutting-edge work with them as a "layman", not a "specialist." This also determines how good these closed-source, private foundat...

Posts

Wikitag Contributions

Comments