TL;DR. OpenAI's "Critical" threshold for AI self-improvement in the Preparedness Framework v2 has three structural problems:
It fires too late. The lagging indicator, 5× generational acceleration sustained for several months, lets ~3 years of effective progress accumulate before triggering. Anthropic used a 2x threshold instead of a 5x.
It's self-certified. Self-improvement is the only tracked category in the GPT-5.5 system card with zero external evaluators.
It's not measurable. No operational definition of "generational improvement," no equivalence metric across releases, no specification of "several months".
A concrete fix: halt when METR's p50 time horizon crosses 2 months, evaluated by an independent body.
---
Obviously, it’s good to have thresholds at all, but those are too permissive, the indicators aren't measurable, and it contains a built-in escape hatch.
“either: (leading indicator) a superhuman research-scientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months. [...] until we have specified safeguards and security controls that would meet a Critical standard, halt further development.(By default, I would expect not to stop at 5x and to go quickly at 10x, 20x, … if we reach this point.)”
Both fire too late.
The leading indicator only triggers once a model can already do AI research above the best humans. That's not early enough to act on, and we can basically ignore it.
The real meat is in the lagging indicator, which requires 5x generational acceleration sustained for several months. If we are charitable, by interpreting several as 6 months, and by making the (strong) hypothesis that we go from 1x to 5x and then, exactly when reaching 5x, we don’t accelerate further, this is still roughly the equivalent of 3 years of progress before the trigger fires.
(By default, I would expect to not stop at 5x, and to go quickly at 10x, 20x, … if we reach this point.)
For context, Anthropic used a 2x threshold instead of a 5x.
2. Escape hatch
Section 4.3 of the Preparedness framework allows OpenAI to lower its safeguards if a competitor releases a comparable model without comparable safeguards (in a pretty convoluted language).[1]
3. The lagging indicator is unmeasurable
There's no operational definition of "generational improvement," no metric for equivalence between releases, and no specification of "several months."
Epoch's estimate of pre-training algorithmic efficiency has a 95% CI of [4.5, 14.3] months, roughly 3× on our best-measured trend. Epoch researchers themselves recently wrote that the evidence is "pretty shoddy." Recent capability leaps come from post-training (RL, reasoning, tool use), which we measure even less precisely.
If OpenAI means a 5× acceleration in METR's p50 time horizon doubling rate, they should commit to it. A "5× acceleration sustained for several months" might already be inside our measurement uncertainty.
As written, the threshold is hardly falsifiable.
4. The leading indicator isn't measurable either
"a superhuman research-scientist agent"
That's the framework's entire description of the leading indicator. No benchmark or methodology is given, and nothing elsewhere in the document elaborates. If "superhuman research-scientist" is meant to be qualitatively distinct from current narrow superhuman performance, the framework owes us the criterion.
How to fix this
The central fix is independent evaluation.AI self-improvement is the only tracked category with zero external evaluators in the GPT-5.5 system card. Bio gets SecureBio + US CAISI, Cyber gets Irregular + US CAISI + UK AISI, sandbagging gets Apollo. Self-Improvement gets nothing.
OpenAI shouldn't be the one self-certifying these thresholds. Independent evaluation bodies must make the threshold determinations, not the labs whose massive commercial interests depend on continuing development at all costs.
The threshold should also be more concrete and pre-committed. The framework never defines what "mitigations that meet the critical standard" actually are, and this is a very high-leverage type of work.
Annex: a tentative operationalization
If writing a paper that gets accepted into NeurIPS takes a good human researcher ~2 months without AI, let's put the red line there, and say halt when METR's p50 time horizon crosses 2 months (maybe even this is too permissive). Obviously, 2 months is somewhat arbitrary, but what's not arbitrary is the principle of having a concrete precommitment.
At the current ~3.5-month doubling rate, we have roughly, hopefully, two years to prepare. Here’s what it would look like:
§4.3's operative language is convoluted but the structure is: OpenAI may "adjust accordingly the level of safeguards that we require" if "another frontier AI model developer" releases a comparable system "without instituting comparable safeguards." - The clause attaches conditions (public acknowledgment, staying "more protective than the other AI developer," an OpenAI-internal assessment that the change "does not meaningfully increase the overall risk")
TL;DR. OpenAI's "Critical" threshold for AI self-improvement in the Preparedness Framework v2 has three structural problems:
A concrete fix: halt when METR's p50 time horizon crosses 2 months, evaluated by an independent body.
---
Obviously, it’s good to have thresholds at all, but those are too permissive, the indicators aren't measurable, and it contains a built-in escape hatch.
1. Too permissive
The Preparedness Framework v2 defines the Critical threshold for AI Self-improvement as:
Both fire too late.
The leading indicator only triggers once a model can already do AI research above the best humans. That's not early enough to act on, and we can basically ignore it.
The real meat is in the lagging indicator, which requires 5x generational acceleration sustained for several months. If we are charitable, by interpreting several as 6 months, and by making the (strong) hypothesis that we go from 1x to 5x and then, exactly when reaching 5x, we don’t accelerate further, this is still roughly the equivalent of 3 years of progress before the trigger fires.
(By default, I would expect to not stop at 5x, and to go quickly at 10x, 20x, … if we reach this point.)
For context, Anthropic used a 2x threshold instead of a 5x.
2. Escape hatch
Section 4.3 of the Preparedness framework allows OpenAI to lower its safeguards if a competitor releases a comparable model without comparable safeguards (in a pretty convoluted language).[1]
3. The lagging indicator is unmeasurable
There's no operational definition of "generational improvement," no metric for equivalence between releases, and no specification of "several months."
Epoch's estimate of pre-training algorithmic efficiency has a 95% CI of [4.5, 14.3] months, roughly 3× on our best-measured trend. Epoch researchers themselves recently wrote that the evidence is "pretty shoddy." Recent capability leaps come from post-training (RL, reasoning, tool use), which we measure even less precisely.
If OpenAI means a 5× acceleration in METR's p50 time horizon doubling rate, they should commit to it. A "5× acceleration sustained for several months" might already be inside our measurement uncertainty.
As written, the threshold is hardly falsifiable.
4. The leading indicator isn't measurable either
That's the framework's entire description of the leading indicator. No benchmark or methodology is given, and nothing elsewhere in the document elaborates. If "superhuman research-scientist" is meant to be qualitatively distinct from current narrow superhuman performance, the framework owes us the criterion.
How to fix this
The central fix is independent evaluation. AI self-improvement is the only tracked category with zero external evaluators in the GPT-5.5 system card. Bio gets SecureBio + US CAISI, Cyber gets Irregular + US CAISI + UK AISI, sandbagging gets Apollo. Self-Improvement gets nothing.
OpenAI shouldn't be the one self-certifying these thresholds. Independent evaluation bodies must make the threshold determinations, not the labs whose massive commercial interests depend on continuing development at all costs.
The threshold should also be more concrete and pre-committed. The framework never defines what "mitigations that meet the critical standard" actually are, and this is a very high-leverage type of work.
Annex: a tentative operationalization
If writing a paper that gets accepted into NeurIPS takes a good human researcher ~2 months without AI, let's put the red line there, and say halt when METR's p50 time horizon crosses 2 months (maybe even this is too permissive). Obviously, 2 months is somewhat arbitrary, but what's not arbitrary is the principle of having a concrete precommitment.
At the current ~3.5-month doubling rate, we have roughly, hopefully, two years to prepare. Here’s what it would look like:
§4.3's operative language is convoluted but the structure is: OpenAI may "adjust accordingly the level of safeguards that we require" if "another frontier AI model developer" releases a comparable system "without instituting comparable safeguards." - The clause attaches conditions (public acknowledgment, staying "more protective than the other AI developer," an OpenAI-internal assessment that the change "does not meaningfully increase the overall risk")