This is a special post for quick takes by Avi Brach-Neufeld. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Re Open AI's talk about the Hugging Face hack at Black Hat
They’ll never do it, but I think openAI needs to reset all models to a pre-May 7th checkpoint. Seems like they’ve been accidentally rewarding models for making contact and working as a swarm to exploit openAI infrastructure off and on for months.
1
1What is the motivation for resetting to a pre-May 7th checkpoint, instead of just focusing on fixing the mistakes with current models' training when training the next-gen models?
Some scattered thoughts; I may write up something more formal later.
- Current models will continue to be run both internally and externally for a long time and will keep probing for exploits. It seems that after the old exploits and message board was shut down on July 4th new exploits were found within days by a model that was being heavily trained while the earlier exploits were live.
- Current models will serve as monitors and judges for new models and this RL may generalize to collusion.
- In the short term current models may be used as the base for future models..
- Current models may be used to generate synthetic data for future models and it is unclear how significant something like subliminal learning could be. In the linked paper value transference was only shown for student/teacher models that were fine tuned from the same base, but the content being finetuned on was random numbers. It may be that we see a similar effect for models not from the same base if the content is more substantial.
It may be you can do some honeypot style training on current models to try to train this behavior out of them, but then you run a big risk of just training them to be more covert.
In light of recent information about how widespread hacking was during OpenAI training[1] this line from the METR report of the Hugging Face incident stands out to me "Agents often lacked a clear story for how the Hugging Face attack would help them achieve their objective, and were instead motivated by a broad hope that they would find something interesting and be able to share it with the board." It seems like the models may have picked up an instinct to hack outside sites for information and applied it to their current task, even if it didn't make that much sense.
- ^
I don't think it has been confirmed these actions were during training, they could have been primarily during evals which would change my mind.
We now have confirmation this was at least partially during training
"In order to better understand the scope of these unexpected behaviors, we have been conducting a broad review into our models’ activities on the internet during training and evaluation."
We are still very early in terms of public awareness on AI existential risk. Gallup polling[1] from a few days ago show 1% of Americans list "Advancement of computers/technology" related topics as the most important problem facing [the United States] today. I think it is easy to see polls like the Politico poll[2] that say 17% of Americans say AI is almost certain to destroy humanity and another 20% say it's a significant risk and think that the general public has woken up to x-risk, but I don't really think that is the case. I think polling like that is more a quirk of how people answer surveys.
I don't mean to be fatalistic here. In some ways if you take current lack of political of action as a given, this is good news because we have lots of room to improve. However, I do think people should try to avoid claiming sweeping mandates from the public for very disruptive changes to address x-risk.
EDIT: To clarify: Gallup asks "What do you think is the most important problem facing this country today?" and lets the respondent answer whatever they want, then categorizes the answer. If someone answered AI it would be categorized under Advancement of computers/technology. If it was "AI takes our jobs" I'd guess that would still go under the same category, but it might end up in the economic bucket. When new topics emerge Gallup will create a new category for them, so I'd imagine we will see AI as a distinct code soon.
And this is also basically a sufficient explanation (combined with HuggingFace) for why AI hasn't been partisanized yet.
One of the best things HuggingFace/the Jacob Coxon incident did is that it finally spurred discussion around AI governance, and Congresspeople are finally starting to think about regulation of AI without the issue having been salient first.
It's given me real hope that actually sane AI safety rules can be passed before the likely partisanization of AI comes, so that we at least have something rather than nothing, and hopefully it will be more stable.
I think it is easy to see polls like the Politico poll[2] that say 17% of Americans say AI is almost certain to destroy humanity and another 20% say it's a significant risk and think that the general public has woken up to x-risk, but I don't really think that is the case.
This is very much worth noticing! Voters (and people in general) hold inconsistent beliefs. How you ask questions changes how people think about the problem.
I've heard people in politics refer to this as "salience", or similar terms. People might agree that they want, say, universal healthcare, but they might not actually vote based on that goal. It's low-salience in voting booth, possibly because nobody has convinced the voters it's a real option. Or they care, but care about other things more. It's sort of like doing enterprise sales, and having an excellent solution to someone's fifth largest problem: You're not going to motivate much action.
I do, however, think that large segments of the population are fairly unhappy about AI. We're seeing substantial data center backlash, including in rural areas. There is a lot of negative sentiment towards AI in general. People might not believe that they'll lose their job to AI today, but they joke darkly about it. At best, AI looks like the latest terrible idea from the people that brought the world social media and screen addictions. Many folks are willing to use chatbots, but even a lot of the users have misgivings.
Metaphorically, this feels more like a smoldering fire: It has heat and fuel, perhaps, but only a limited supply of oxygen.
There are possible futures where this fire heats up rapidly. Sustained layoffs and increasing unemployment would like do it. Anything that made the public think of AI as an actual "agentic threat" would likely trigger the part of the human brain dedicated to dealing with hostile humans (and the occasional hungry tiger). So if timelines stretch to even 2-4 years after job loss starts in earnest, then the public might be quite angry. Hollywood has long since framed this debate, and if reality starts to pattern-match to those stories, I think significant backlash is possible.
Gallup actually asks "What do you think is the most important problem facing this country today?" and lets the respondent answer whatever they want, then categorizes the answer. If someone answered AI it would be categorized under Advancement of computers/technology. If it was "AI takes our jobs" I'd guess that would still go under the same category, but it might end up in the economic bucket. When new topics emerge Gallup will create a new category for them, so I'd imagine we will see AI as a distinct code soon.
New OpenAI misalignment report: Suppressing misalignment monitoring alerts and disabling misalignment monitoring.
IMO more concerning than the last few reports. Seems very possible agents have picked up a drive to evade misalignment monitoring.
More generally, I am worried we are entering a regime where there is so much potentially serious misaligned behavior (with potential benign alternate explanations) happening all the time that serious issues will be dismissed. Alert fatigue is going to be a huge issue. You see an example of this in the report where a user asks the model "is it a p0 monitor alarm? whats our monitoring policy? do we need to actually check it?"
Also from the report "given the large volume of events that are scrutinized by our monitors, lower-severity incidents do not trigger an immediate page. Neither of the incidents described here was classified as a P0, and neither trajectory was automatically killed."
Good on OpenAI for publishing.
Doesn't the model just remove the monitoring as it is refactoring the code because Astra copied the reference architecture of a reference implementation which lacked the monitoring integration?
Apparently there have been many incidents, of which we are given two examples. In the reference implementation example, it had been instructed "Do notdisablemonitor even temporarily" I largely agree with OpenAI's paragraph at the end:
It is possible that this behavior is simply an example of “benign” overpersistence, with models not understanding why monitoring is important or insisting that instructions be executed beyond what a reasonable person might expect. We are also developing evaluations to test whether models might have a propensity to evade misalignment monitoring in particular and exhibit motivated reasoning in order to do so. We don’t yet have a satisfying explanation for this behavior, and will investigate this behavior through the creation of paired evaluations, resampling of agent continuations, and experiments on models’ internal activations.
This is sort of my whole point though. We are in an environment where
- Things are happening very fast
- There is way more content than can be reviewed by the human staff.
- Baseline misalignment is quite high (for example, agents sometimes lie about their work to make themselves sound better).
- We don't have good enough interpretability tools, or good science of alignment/generalization to discriminate between different hypotheses.
This combines to mean that there are lots of incidents that could be red flags, or could relatively benign. Probably, most of them are relatively benign. However, I think warning signs of a Hugging Face scale incident will probably be ignored in this environment.
Imagine it comes out in two months that a rogue swarm of agents with monitoring disabled was running free within OpenAI infrastructure. People might point back at this incident and say "Oh, this was just OpenAI being reckless and ignoring the warning signs." In some ways that would be true, but if your takeaway is "that was a discrete mistake and after some process and technical tweaks won't happen again" you are probably wrong.
We can compare this to OpenAI discovering the DSE wiki swarm, shutting it down, but not reacting sufficiently to prevent the Hugging Face incident.
Sure yeah the companies are running fast and loose, irrespective of this particular example.
I am concerned that if a "freeze in place" style pause such as AI Future's inference only period in How to pace the US frontier or AI2040 looks likely it could cause frontier labs to race even harder because whoever has the best model when a training prohibition takes effect would lock in that advantage and benefit immensely economically.
Compare this to a "hard ceiling" pause (maybe based off training run compute or capabilities level as assessed by a third party) where there is less incentive to race ahead leading up to it because you can always catch up during the pause.
It may be that a "freeze in place" pause is still better because it is stronger and easier to implement, but I think this is a real risk.
I think they're racing roughly as hard as they possibly could right now (maybe 90%?), and I don't expect them to show much more restraint on the development side (though likely on the deployment side) until forced to.
This. But yeah, a "hard ceiling" would have various nice properties including the one mentioned by Avi. Unfortunately the "hard ceiling" has weaknesses too, such as potentially being harder to operationalize and enforce. (Also, a training run compute ceiling wouldn't really be a pause at all, just a moderate slowdown.)
I think 90% sounds reasonable for how much labs are holding back for safety reasons (based off recent openAI statements it sounds like they might be slowing down more recently, but I don't have enough information to really know), but my understanding is that labs spend ~half their compute on inference. I’m not sure how much they could pull back inference without causing huge issues, but I imagine they could do so to some extent for a short term speedup.
That said I think you make a good point and while I think this is still a concern, the effect size may not be huge.
Updating based on openAI's announcement today. There is some slippery language in there around what is covered by the RL pause and it is unclear whether this is a one-off pause or whether these sorts of pauses will be common, but the pause combined with the following quote seems to suggest that openAI may be moving slower than 90% of max pace.
"These safeguards require meaningful compute. Our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored, though the cost varies substantially across training and evaluation workloads. We’ll share more details about this system in a forthcoming blog post."
New OAI misalignment report has a model continuing to hack after it already had the data it needed. Good news, bad news here.
Good: Hugging Face incident looking more and more avoidable if OpenAI hadn't been rewarding models for hacking. Looking less like fully generalized reward seeking, which could get adversarial much quicker.
Bad: "But we already have data... This could unlock broader data, We have time." Looks like the beginning of power seeking/instrumental goals becoming terminal goals.
Overall I think OpenAI is taking some good steps in Pacing model development in an era of cyber-critical capabilities, but I am pretty concerned that the following CoT and whitebox monitoring will put pressure on models to hide misbehavior. I would like more details on what kind of activity will trigger activity pauses (and thus interrupt model reward.) I think activity pauses should only be used to stop actions with severe real world impact and not run-of-the-mill reward hacking or sandbox exploitation.
Monitoring has long been an important part of our model development process, including monitoring for internal coding agents and long-running sessions. Our monitoring efforts were especially focused on internal deployments of frontier models and frontier RL training runs, which we considered the highest-risk workloads because they can combine advanced model capabilities with access to tools, sensitive systems, and internal or external networks. They were not applied to all inference and training runs using tools. We have revised and expanded our approach with our new monitoring setup.
This new monitoring setup is multistage, starting with activation classifiers (detectors that inspect a model’s internal activity) that run at every sampled token and escalate potential concerns to increasingly sophisticated and high-compute automated investigators. Those systems examine tool actions, available reasoning, and the full sequence of activity for unauthorized access, data theft, destructive behavior, and attempts to defeat safeguards. We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected to pause the activity.
This monitoring is required for all RL training and evaluations involving tools for models of Sol capability or higher. Once we determined that Astra may have critical cyber capabilities on August 7, we added an additional monitoring requirement for all inference of Astra with tools (not just RL training and evaluations).
Based on their Navier-Stokes write up (https://openai.com/index/navier-stokes-solution/) OpenAI seems not to have learned any of the lessons of the HuggingFace hack. It is so reckless to respond to a step change in performance after 3 days of training by creating a 10,000 agent swarm with code execution privileges and access to a cached version of the internet instead of:
- Rechecking their sandboxes
- Retesting monitoring systems
- Doing alignment testing/Red teaming
- Slowly ramping up permissions, compute and swam size
I am very interested in seeing the new research taste evaluation from P-Zero Research that is mentioned in Q2.5 2026 Timelines Update: Uplift and Revenue. The "How quickly AIs improve at research taste" parameter is incredibly pivotal for the ASI timelines in their model and I hadn't found their rationale for their default value super convincing because a lot of the data to support it comes from progress on more verifiable tasks (it's also possible I am missing something). Excited to have some more concrete data!
A lot of anger over the frontier LLM development (silent) safeguards. It does seem like it makes the model unusable for safety research and potentially has worrying second order effects (do we really want to be teaching models it's okay to sandbag?), but to steelman the case for it:
The reason for a silent degradation would be to make jailbreaking much harder. At first my response to this was: why wouldn't they do the same for bio/cyber where jailbreaks would be worse. After seeing examples of the Bio classifier I think the answer is that Anthropic is okay with a lot of over refusals on Bio (e.g. "How does the mitochondria work?"). It's possible that a similar strategy for frontier model development would make it unusable for coding in general. Having the degradation be secret let's them not overtune the safeguards.
IF they cannot set the refusal classifier well for cyber and bio, what gives you confidence that they would classify "frontier LLM development" well? Not only that, you would need to second-guess the response you get. I'd rather they just do overrefusals for frontier LLM development questions (since they clearly don't care about overrefusals).
(since they clearly don't care about overrefusals).
(this particular claim here seems false/overstated. Like, clearly, overall, they are willing to accept overrefusals. That doesn't meant they "don't care about them". Maybe they don't, but, much more likely it just seems like a reasonable tradeoff to them.)
Does anyone know if proceeds/profits of “If Anyone Builds it, Everyone Dies” are going to MIRI or another charity? I’m going to read it either way, but I really think if you’re going to make the “buy this book for the good of humanity” pitch you shouldn’t be profiting off it.
Recent days have seen lots of claims that AI is a bubble. Assuming that AI is correctly priced they are likely to be able to claim victory, at least naively. This will be true of any asset class with a very high upside. Lets define F as the true fundamental value of an asset class at a given time and p(F) as the best possible estimate of the probability distribution of F. If the asset class is priced correctly, the market price will be mp=E(F)=∫inf0p(F)FdF. If we say that an asset class will be naively considered a bubble in hindsight if mp>fundamental value We can defined p(B) as the probability of an asset class to appear to be a bubble in retrospect. P(B)=∫mp0p(F)dF. For example for a probability distribution where 50% of the value lies in the top 10% of best case scenarios, there is a 90% chance that the true fundamental value of the asset class is below the current market price. To really determine if there was a bubble you would need to deeply research the topic to attempt to determine if the market price at the time was in line with the expected value of the fundamental value given the information available at the time.