Rauno's Shortform

Rauno Arike

This is a special post for quick takes by Rauno Arike. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Why do frontier labs keep a lot of their safety research unpublished?

In Reflections on OpenAI, Calvin French-Owen writes:

Safety is actually more of a thing than you might guess if you read a lot from Zvi or Lesswrong. There's a large number of people working to develop safety systems. Given the nature of OpenAI, I saw more focus on practical risks (hate speech, abuse, manipulating political biases, crafting bio-weapons, self-harm, prompt injection) than theoretical ones (intelligence explosion, power-seeking). That's not to say that nobody is working on the latter, there's definitely people focusing on the theoretical risks. But from my viewpoint, it's not the focus. Most of the work which is done isn't published, and OpenAI really should do more to get it out there.

This makes me wonder: what's the main bottleneck that keeps them from publishing this safety research? Unlike capabilities research, it's possible to publish most of this work without giving away model secrets, as Anthropic has shown. It would also have a positive impact on the public perception of OpenAI, at least in LW-adjacent communities. Is it nevertheless about a fear of leaking information to competitors? Is it about the time cost involved in writing a paper? Something else?

I think the extent to which it's possible to publish without giving away commercially sensitive information depends a lot on exactly what kind of "safety work" it is. For example, if you figured out a way to stop models from reward hacking on unit tests, it's probably to your advantage to not share that with competitors.

1thinking1y

From my experience, there are usually sections (for at least the earlier papers historically) on safety for model releasing. * PaLM: https://arxiv.org/pdf/2204.02311 * Llama 2: https://arxiv.org/pdf/2307.09288 * Llama 3: https://arxiv.org/pdf/2407.21783 * GPT 4: https://cdn.openai.com/papers/gpt-4-system-card.pdf Labs release some general paper like (just examples): * https://arxiv.org/pdf/2202.07646, https://openreview.net/pdf?id=vjel3nWP2a etc (Nicholas Carlini has a lot of paper related to memorization and extractability) * https://arxiv.org/pdf/2311.18140 * https://arxiv.org/pdf/2507.02735 * https://arxiv.org/pdf/2404.10989v1 If we see less contents in these sections, one possibility is increased legal regulations that may make publication tricky (imagine an extreme case, companies have sincere intents to produce some numbers that is not indication of harm yet, these preliminary or signal numbers could be used in an "abused" way in legal dispute, may be for profitability reasons/"lawyers need to win cases" reasons). To remove sensitive information, it would comes down then to time cost involved in paper writing, and time cost in removing sensitive information. And interestingly, political landscape could steer companies away from being more safety focused. I do hope there could be a better way to resolve this, providing more incentives for companies to report and share mitigations and measurements. As far as I know, safety tests usually are used for internal decision making at least for releases etc.

1thinking1y

Could you elaborate what do you mean by "mundane" safety work?