Daniel Tan's Shortform

Daniel Tan

This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

In a recent job posting, OpenAI describes what sounds like a "notice problem, apply shallow fix" approach to addressing misalignment risk (emphasis mine)

Training: Create training methods, environments and data that teach agents to make better decisions in consequential situations. We turn real-world failures into training signals that prevent similar incidents, and identify precursor behaviors and mitigations to address emerging risks.

I find this pretty concerning, since the most straightforward reading is "we'll take Huggingface and train our models so that there wouldn't be future Huggingface incidents, but in a way which wouldn't really fix the underlying misalignment." This would be a recapitulation of many mistakes described in "alignment engineering vs misalignment science". OTOH there are some signs that they aren't going to do just that, e.g. "identify precursor behaviors" sounds relatively more promising.

Hard to know how to update on this I guess. Still seems worth keeping in mind

I don't know much about OpenAIs hiring patterns, so please correct me if I'm wrong; but I would probably expect researchers who are working on deeper more fundamental alignment work to relatively more senior and hired through different pathways on average (in industries which I'm more familiar with it's very rare for senior positions to be openly advertised at all). I think there's still a conservation of expected evidence effect, and updating minorly makes sense; but I think it should be very minor.

1Thomas Jiralerspong8h

This aligns with what people at OpenAI have told me. In addition, any alignment progress they make is embargoed because alignment is increasingly the bottleneck to releasing models.

5Thomas Kwa14h

i think this shouldn't be a big update. some thoughts: * OpenAI was always going to try to solve alignment problems in similar ways that they solve capabilities problems. Most of the variance is whether they do misalignment science in addition to this, so one should look for the presence of those job postings as well as public statements on investment into better misalignment measurement, whether these measurements will be held out, model organisms work, etc * I'm only like 50% sure this is about huggingface. according to a colleague, "severe unintended outcomes from increasingly capable AI agents" could be about agents deleting files or spending $1000 of your money or something. * Fixing training signals can mean a wide range of things. Fixing RL environments to not reward egregious misalignment addresses a root cause of incidents, which seems fine. * However, this isn't much better long-term than something worse (e.g. continuing to reward egregious misalignment but also punishing it in a way that doesn't generalize) (except maybe evidence about openai's attitude), because if it doesn't generalize, they'll just keep having incidents until they implement fixes that generalize at current capability levels.

Daniel Tan's Shortform

2

Daniel Tan's Shortform

2