OpenAI has announced their first round of the embedded assessors they promised in September. Sam Altman said the assessors would have “employee-like” access. I think that a good version of embedded assessment could be extraordinarily helpful for reducing risk. As it stands, I think these assessments will leave most of the value on the table.
A lot of the problem is in the (frustratingly vague) details of these agreements. If we want future agreements to be better, we’re going to have to read into the details, so I'm sharing my thoughts on the details here. I hope people will take the time to dig into the weeds and form their own views.
All four of the embedded assessment agreements OpenAI announced seem very narrow in scope. The assessors have pointed out how limited the scope is:
We need embedded assessors to create holistic assessments of whether the company's activities they're about to embark on are risky.
Specifically:
In an ideal world, we would already be in a state where multiple third-party organizations get actual employee-level access into the companies, including the Slack, relevant documents, ability to resample transcripts, deployment simulation tools, finetuning access, access to training code, access to internal tools and infrastructure used by researchers, appropriate process transparency guarantees, and no hard date or scope cutoffs. We are very far away from that world.
At least one of Apollo or METR should have access to Slack and the codebase. They should also be able to assess every model and to hear about and assess plans for training runs before they begin.
I’m also worried about the restrictions on employee interviews. One of the most important ways an assessor can understand if they have all the relevant information, get a high-level picture, and discover unknown unknowns is via talking to employees.
METR can only “use OpenAI staff interviews to guide [their] work,” [source] and says “our reports cannot quote or cite individual views.” In effect, if something an employee says to METR is not directly captured by “transcripts, monitor records, and snapshots of the model throughout RL,” then METR will not be able to tell the public what was said. This seems extremely restrictive, as risk-relevant evidence might take many different shapes and forms.
Apollo can “publish information based on employee discussions” [source] if “OpenAI provides formal attestation, or we can independently establish the finding.” Again, these measures seem very draconian and will predictably incentivize OpenAI employees to raise fewer worries.
OpenAI is just not meeting the level of ambition needed to lay the technical foundations for pacing or risk assessment. They haven’t even announced something equivalent to the recent AI R&D acceleration assessment that METR did with Anthropic, and that’s lower than the bare minimum to be considered “trying to prepare for pacing” at this point. It will soon be extremely important to be able to verify whether a company is adhering to a pacing regime (e.g. through publicly reporting RSI metrics and compute allocation), and OpenAI does not seem to be setting the foundations for that.