This is a special post for quick takes by anaguma. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Tristan Buckmaster, a mathematician at NYU's Courant Institute, says OpenAI researchers told him that an internal model had produced a proof of finite-time blowup for the Navier-Stokes equations with smooth forcing. If correct, this would resolve the Millennium Prize problem in the negative.
Buckmaster and his collaborator Levent Alpöge (who works at Anthropic) had spent a year working on this problem, with the help of Claude and Codex, and had released Lean-verified proofs for related problems. He was concerned that OpenAI might have been training on their Codex sessions. He described a call with Sébastien Bubeck and another OpenAI mathematician as follows:
...I asked whether the model had been trained on, or had access to, our sessions in Codex, into which we had been putting all our drafts for the whole of this project. I was told the model did not look up user data. I asked again, about training, and I did not get an answer.
Two proposals were offered to me. The first was that we post our Euler result, and that OpenAI post its Navier-Stokes result the next day. The second was that, after posting Euler, I alone write a paper presenting the Navier-Stokes result, acknowledging that an
1I find it hard to believe that OAI looked at the logs, but it seems clear that they heard that Buckmaster/Alpöge were close and decided to throw the kitchen sink at trying to beat them to it, even though it doesn't appear to be an Anthropic project (I've seen people cast doubt on this, but e.g. the Jacobian conjecture resolution was announced via a tweet from Alpöge with no follow up blog post from Ant, and it looks like Buckmaster openly told OAI that they were using their models to work on the problem).
I also find it hard to think of an innocent explanation for trying to get Alpöge removed as a coauthor.
5
4i don't know about the details of sebastien and buckmaster and levent's conversation. but i don't think openai as an organization has done anything obviously untoward here?
2This is not how a frontier lab should be interacting with the math community.
Why would we expect them to behave any differently? They want to be first, and they don't want to share. In all things, not just math, the aim has been to reach the finish line, regardless of whether the outcome is good, useful, truthful, safe, honest, or thoughtful.
Even if the math community decided to punish them, what force could they muster? The power is in their hands now, and academia is increasingly weakened. When frontier labs use up the remainder of the math community and discard them like rubbish, I don't think there will be much surprise.
[This comment is no longer endorsed by its author]
3
21
Just trying to model the possible aftermath of this.
Even if no plagiarism occurred, if training on chats contributed meaningfully to OAI's result, there's still a strong disincentive for any R&D firm against using OAI products. Plus, OAI's behavior (if true) is extremely off-putting. The general culture around knowledge production would shift toward hoarding knowledge, which I think would be a huge loss to progress. I also suspect that many people put all AI companies into the same mental bucket, to where the actions of one heavily influence how they feel about the others. So, even if OAI is the one who screwed up, I think people would broadly distrust any AI at all.
I think this case has much greater potential for igniting anti-AI sentiment than previous complaints about artistic plagiarism. The analogous case for AI art is a human studying an artist and being inspired. We already see that as inspiration, not plagiarism, so the average person doesn't immediately see AI art as plagiarism unless there's an extreme similarity. But in this case, the closest analogy is an academic finding someone's unpublished work, using that to produce their own, and then rushing to publish to seize more of the credit. Even if OAI never saw the previous work, I could see this type of narrative being pushed among the general public. OAI's behavior is weird enough to weaken credibility, so it seems likely that some people will dismiss any plausible explanation in favor of a conspiracy narrative. The HF incident is perfect timing for OAI to appear as general liars.
I'm immensely curious as to how this plays out. X is already buzzing with accusations of intentional plagiarism. Please let me know if I'm missing anything in my guesses.
I think Leopold's firing from OpenAI in April 2024 provides some relevant context for the recent firing of other safety researchers, and in particular what the company considers leaking. From the Dwarkesh Podcast:
...Dwarkesh Patel 02:31:26
Now Jan and Ilya have left, and the team itself has dissolved. You were the first person who left or was forced to leave. The Information reported that you were fired for leaking. What happened? Is this accurate?
Leopold Aschenbrenner 02:31:43
Why don’t I tell you what they claim I leaked, and you can tell me what you think. OpenAI claimed to employees that I was fired for leaking. I and others have pushed them to say what the leak was. Here’s their response in full: Sometime last year, I had written a brainstorming document on preparedness, safety, and security measures needed in the future on the path to AGI. I shared that with three external researchers for feedback. That’s the leak.
For context, it was totally normal at OpenAI at the time to share safety ideas with external researchers for feedback. It happened all the time. The doc had my ideas. Before I shared it, I reviewed it for anything sensitive. The internal version had a reference to a futu
3
1Other info from the announcement worth mentioning:
- was a general model, not specialized, they were just testing it on random Erdos problems
- key trick seemingly was applying algebraic number theory to geometry in an unexpected way
1I found the remarks on the problem by the mathematicians OpenAI brought in to check the proof very enlightening:
https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29ad73/unit-distance-remarks.pdf
It seems as if this is a significant achievement, but also that this conjecture was of most interest to mathematicians because it was thought to be true, and it was believed that proving it would require new and interesting tools. Instead the model proved it to be false using less interesting mathematics. It seems like another example (iirc, the Frontiermath open problem solved by GPT 5.4 was similar?) where models not having the biases of most mathematicians (in this case, trying to prove the conjecture rather than disprove it) was very helpful.
16
The paper provides the original output the model gave before any rewriting, starting on page 3. I was kind of expecting a big mess, but it's really not. It's pretty short by the standards of tricky proofs. Two and a half pages, most of it text.
A 200-layer non-looped Transformer has no more serial depth than a 50-layer 4x looped transformer. I don't think there's a principled reason for calling the activations in the residual stream "neuralese" in the second case but not the former. So the mere fact of recurrence doesn't mean that it's now false that "[l]ong serial chains of cognition must pass through the chain of thought," per your link.
(IMO "neuralese" means a bunch of different things that are only kinda loosely connected. An LSTM LLM is "neuralese" if recurrence over weights is all it takes.)
It's almost certainly possible to limit the degree to which CoT interpretability or monitorability is compromised by loops by limiting the number of loops.
On the other hand, it is true that this might lead to degredation of CoT monitorability in the future, if it works to pump up the number of loops indefinitely and they decide to do so, or if RL interacts with looped transformers in a weird way. I don't think the mere fact of this architectural choice = "CoT monitorability is compromised."
47
Some OAI safety people seem to be denying this is true, though I lack the technical knowledge to know if this is a flat denial or if there are other ways it could be true (e.g. the technique could be used to obfuscate reasoning, but OAI have implemented it in a way that means that it doesn't for now)
https://x.com/MicahCarroll/status/2095023282051563835
https://x.com/tomekkorbak/status/2095031132781961346
3
What happened to the old rumors about Mythos/Fable being looped?
3
I heard rumors fable also uses looped transformers.
1
Reading some papers, I could not actually find a reference that says that a looped model outperforms its “weight-untied” variant for any given amount of training tokens.
The advantage papers talk about is an increase in memory efficiency since you can use a weight multiple times, not an advantage in training data efficiency compared to an “untied” model. In the Vladimir Nesov evaluation strategy I don’t think it even reduces latency for frontier models unless you can keep a layer or two in fast memory, which I don’t think you can (or can you? I don’t know GPUs well enough for that).
SSI announced that they are scaling up their work. Surprisingly, they also shared some of their research with Nvidia. Perhaps this means they will release a product after all.
...Ilya Sutskever’s Safe Superintelligence Inc. and NVIDIA Announce Long-Term Strategic Partnership
Access to NVIDIA’s Best-in-Class Vera Rubin Systems Expands SSI’s Compute by an Order of Magnitude
SANTA CLARA, Calif. and PALO ALTO, Calif., July 27, 2026 (GLOBE NEWSWIRE) -- Safe Superintelligence Inc. (SSI) and NVIDIA today announced a long-term partnership to rapidly accelerate SSI’s strategic growth. NVIDIA has additionally made an investment in SSI.For SSI, NVIDIA’s substantial investment combined with access to the next-generation, best-in-class NVIDIA Vera Rubin platform will allow SSI to increase its compute by an order of magnitude. The two companies will also collaborate on the technical advancement of NVIDIA’s current and future compute platforms, leveraging SSI’s unique insights into the future of AI.
For the last two years, SSI has been quietly advancing a new research direction to unlock a powerful and robustly aligned artificial intelligence. NVIDIA entered this partnership to accelerate SSI’s next st
36
In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly
1. Timelines end up being long
2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x'ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM's are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don't release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I think they'll release an underwhelming product that is neither superintelligent nor safe, mostly as a bid for relevance.
I don't think Ilya is any better than Sam, Dario, Demis, or Elon. They're all AI company CEOs, which is a morally compromising thing to be. They're all trying to build superintelligence, which is a very evil thing to try to do.
63Looking over the comments, some of the most upvoted comments express the sentiment ththat Yudkowsky is not the best communicator. This is what the people say.
1I'm afraid the evolution analogy isn't as convincing an argument for everyone as Eliezer seems to think. For me, for instance, it's quite persuasive because evolution has long been a central part of my world model. However, I'm aware that for most "normal people", this isn't the case; evolution is a kind of dormant knowledge, not a part of the lens they see the world with. I think this is why they can't intuitively grasp, like most rat and rat-adjacent people do, how powerful optimization processes (like gradient descent or evolution) can lead to mesa-optimization, and what the consequences of that might be: the inferential distance is simply too large.
I think Eliezer has made great strides recently in appealing to a broader audience. But if we want to convince more people, we need to find rhetorical tools other than the evolution analogy and assume less scientific intuition.
14
That’s a bummer. I’ve only listened partway but was actually impressed so far with how Eliezer presented things, and felt like whatever media prep has been done has been quite helpful
2
I am a fan of Yudkowsky and it was nice hearing him of Ezra Klein, but I would have to say that for my part the arguments didn't feel very tight in this one. Less so than in IABED (which I thought was good not great).
Ezra seems to contend that surely we have evidence that we can at least kind of align current systems to at least basically what we usually want most of the time. I think this is reasonable. He contends that maybe that level of "mostly works" as well as the opportunity to gradually give feedback and increment current systems seems like it'll get us pretty far. That seems reasonable to me.
As I understand it, Yudkowsky probably sees LLMs as vaguely anthropomophic at best, but not meaningfully aligned in a way that would be safe/okay if current systems were more "coherent" and powerful. Not even close. I think he contended that if you just gave loads of power to ~current LLMs, they would optimize for something considerably different than the "true moral law". Because of the "fragility of value", he also believes it is likely the case that most types of psuedoalignments are not worthwhile. Honestly, that part felt undersubstantiated in a "why should I trust that this guy knows the personality of GPT 9" sort of way; I mean, Claude seems reasonably nice right? And also, ofc, there's the "you can't retrain a powerful superintelligence" problem / the stop button problem / the anti-natural problems of corrigible agency which undercut a lot of Ezra's pitch, but which they didn't really get into.
So ya, I gotta say, it was hardly a slam dunk case / discussion for high p(doom | superintelligence).
8
The comments on the video are a bit disheartening... lots of people saying Yudkowsky is too confusing, answers everything too technically or with metaphors, structuring sentences in a way that's hard to follow, and Ezra didn't really understand the points he was making.
One example: Eliezer mentioned in the interview that there was a kid whose chatbot encouraged him to commit suicide, with the point that "no one programmed the chatbot to do this." This comment made me think:
[...]
Oh yeah, probably most people telling this story would at least mention that the kid did in fact commit suicide, rather than treating it solely as evidence for an abstract point...
7
Klein comes off very sensibly. I don’t agree with his reasons for hope, but they do seem pretty well thought out and Yudkowsky did not answer them clearly.
6
I was excited to listen to this episode, but spent most of it tearing my hair out in frustration. A friend of mine who is a fan of Klein told me unprompted that when he was listening, he was lost and did not understand what Eliezer was saying. He seems to just not be responding to the questions Klein is asking, and instead he diverts to analogies that bear no obvious relation to the question being asked. I don't think anyone unconvinced of AI risk will be convinced by this episode, and worse, I think they will come away believing the case is muddled and confusing and not really worth listening to.
This is not the first time I've felt this way listening to Eliezer speak to "normies". I think his writings are for the most part very clear, but his communication skills just do not seem to translate well to the podcast/live interview format.
8
Also, according to METR:
[...]
3
Thanks. Does anyone know why METR released the summary but not the full report? They’ve done so in the past, but I couldn’t find it this time.
I'm not sure but the wording in their footnote 1 seems unusually careful:
We think it’s valuable for AI developers to be able to share specific technical details with third parties without this information being shared further, and it’s very reasonable for AI developers to review 3rd-party eval reports to ensure no accidental sharing of sensitive IP.
We had an informal understanding with OpenAI that their review was checking for confidentiality / IP issues, rather than approving conclusions about safety or risk. We did not make changes to conclusions, takeaways or tone (or any other changes we considered problematic) based on their review. We are able to freely publish parts of the evaluation that depended only on information that is now public.
However, we expect some readers will want us to note that OpenAI would have had the legal right to block us from sharing conclusions about risk that depended on non-public information. Given that, this evaluation shouldn’t be interpreted as robust formal oversight or accountability that the public can be relying on METR to provide.
...That being said, we think this evaluation is an excellent step forward and we are very supportive of pr
1A notable section from Ilya Sutskever's recent deposition:
WITNESS SUTSKEVER: Right now, my view is that, with very few exceptions, most likely a person who is going to be in charge is going to be very good with the way of power. And it will be a lot like choosing between different politicians.
ATTORNEY EDDY: The person in charge of what?
WITNESS SUTSKEVER: AGI.
ATTORNEY EDDY: And why do you say that?
ATTORNEY AGNOLUCCI: Object to form.
WITNESS SUTSKEVER: That's how the world seems to work. I think it's very -- I think it's not impossible, but I think it's very hard for someone who would be described as a saint to make it. I think it's worth trying. I just think it's -- it's like choosing between different politicians. Who is going to be the head of the state?
22I think there are a bunch of parts of Plan A which would be difficult for both sides to agree to (e.g. Mutually Assured Compute Destruction). It would be good to have a simplified version (Plan A-) which requires less political will and is less risky than Plan B/C/D. For example:
If there isn’t enough political will for Plan S (a full pause) or Plan A, we could pass an international ban specifically on AIs assisting in AI research and hardware design. It seems like this prevents most of the risks of the intelligence explosion, though the default rate of algorithmic and hardware progress will continue. It might be politically easier for the US and China to agree to this plan since it has fewer economic costs (e.g. doesn’t directly limit GPU production) and verification could be less intrusive. Since neither side may want total research transparency, each could have its AIs periodically audit the other’s logs for signs of violations without revealing secrets. You could also have each side agree to run a lightweight probe/classifier which continually looks for signs of assisting AI research and have it share the results (some early ideas of how to do so here). If both sides agree, it c...
1OpenAI claims to have paused frontier RL training for now. Altman stated on X:
We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.
We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.
We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.
3Just because they haven't drawn that lesson yet doesn't mean this data point won't be remembered and contribute to their eventual drawing of that lesson.
11
Isn't the explanation just that an influential AI blog named GPT 5 his "Research Goblin"?
If anyone survives, no one builds it.
1Inoculation Prompting has to be one of the most janky ad-hoc alignment solutions I've ever seen. I agree that it seems to work for existing models, but I expect it to fail for more capable models in a generation or two. One way this could happen:
1) We train a model using inoculation prompting, with a lot of RL, using say 10x the compute for RL as used in pretraining
2) The model develops strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment
3) In the production environment, we remove the statement saying that reward hacking is okay, and replace it perhaps with a statement politely asking the model not to reward hack/be misaligned (or nothing at all)
4) The model reflects upon this statement ... and is broadly misaligned anyway, because of the habits/drives developed in step 2. Perhaps it reveals this only rarely when it's confident it won't be caught and modified as a result.
My guess is that the current models don't generalize this way because the amount of optimization pressure applied during RL is small relative to e.g. the HHH prior. I'd be interested to see a scaling analysis of this question.
1strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment
Perhaps automated detection of when such methods are used to succeed will enable robustly fixing/blacklisting almost all RL environments/scenarios where the models can succeed this way. (Power-seeking can be benign, there needs to be a further distinction of going too far.)
13
This hinges on questions about the kinds of circuits which LLMs have (I think of these as questions about the population of Logical Induction traders which make up the LLMs internal prediction market about which next token gets high reward).
Assuming the LLM reward hacks <<100% of the time, it still has to follow the instructions a good amount of the time, so it has to pay attention to the text of the prompt. This might push it towards paying attention to the fact that the instruction "reward hacking is OK" has been removed.
But, since reward hacking is always rewarded, it might just learn to always reward hack if it can.
"Richard Sutton rejects AI Risk" seems misleading in my view. What risks is he rejecting specifically?
His view seems to be that AI will replace us, humanity as we know it will go extinct, and that is okay. E.g., here he speaks positively of a Moravec quote, "Rather quickly, they could displace us from existence". Most would consider our extinction as a risk they are referring to when they say "AI Risk".
1As an example of what RSI might be like, I find it helpful to go back to OpenAI's Dota 2 result from 2017:
This slide from Ilya's lecture shows the bot's Trueskill rating[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.
- ^
Similar to Elo in Chess and other games
I was surprised to learn recently that the error bars on the METR time horizon chart are this large. This is probably the most important capabilities benchmark right now[1], but I don't think it's precise enough to be useful for discussions about AI capabilities progress or RSI.
Why hasn't METR added more long-horizon tasks to their benchmark since it was released in March 2025? I think they could probably find funding to do this from the labs or EA donors.
I think they are working on adding new tasks? Not sure. Apparently it's hard. This concerns me greatly too, because basically their existing benchmark is about to get saturated and we'll be flying blind again.
My hope is that the entire AI benchmarks industry/literature will reform itself and pick up the ideas METR introduced. Imagine:
--It becomes standard practice for any benchmark-maker to include a human baseline for each task in the benchmark, or at least a statistically significant sample.
--They also include information about the 'quality' of the baseliners & crucially, how long the baseliners took to do the task & what the market rate for those people's time would be.
--It also becomes standard practice for anyone evaluating a model on a benchmark to report how much $ they spent on inference compute & how much clock time it took to complete the task.
If the industry/literature adopts these practices, then every benchmark basically becomes a horizon length benchmark. We can do a giant metaanalysis that aggregates it all together. Error bars will shrink. And The Graph will continue marching on through 2026 and 2027 instead of being saturated and forgotten.
2115
you should be more uncertain about the METR benchmark's external validity than what these error bars show.
but your baseline uncertainty about key facts about AI progress in general should also often span much more than one order of magnitude between your 2.5th percentile and 97.5th percentile guess. the METR results add a lot of value and I don't think these error bars are a big deal in the scheme of things.
3
Some of that error is correlated between models; they also have versions of the graph with error bars on the trendline and those error bars are notably smaller.
The error bars are also much smaller when you look at the plot on a log-y-axis. Like, in some sense not being able to distinguish a 10-minute time horizon from a 30-minute one is a lot of error, but it's still very distinct from the one-minute time horizon of the previous generation or the 2-hour time horizon you might expect from the next generation. In other words, when you look at the image you shared, the error bars on o4 mini don't look so bad, but if you were only looking at models up to o4 mini you'd have zoomed in a bunch and the error bars on o4 mini would be large too.
Also note that to cut the size of the error bars in half you'd need to make ~4x as many tasks, to cut it by 4x you'd need ~16x as many tasks. And you'd need to be very confident the tasks weren't buggy, so just throwing money at the wall and hiring lots of people won't work because you'll just get a bunch of tasks you won't have confidence in.
Keep in mind the opportunity cost is real though, and the main blocker on orgs like METR usually is more like talent/capacity than money. It would be great if they had capacity for this and you're right that it is insane that humanity doesn't have better benchmarks. But there's a dozen other fires at least that large that METR seems to be trying to address, like RCTs to see if AI is actually speeding people up and risk report reviews to see if AIs are actually safe. Perhaps you think these are less important, but if so I would like to hear that argument.
All that said, my understanding is METR is working on this. I would also love to see this type of work from others!
For what it's worth, I don't think it matters for now, for a couple of reasons:
Most of the capabilities gained this year have come from inference scaling which uses CoT more heavily than pre-training scaling which improves forward passes,though you could reasonably argue that most RL inference gains are basically just a good version of how scaffolding would work in agents like AutoGPT, and don't give new capabilities.- Neuralese architectures that outperform standard transformers on big tasks turn out to be relatively hard to do, and are at least not trivial to scale up (this mostly comes from diffuse discourse, but one example of this is here, where COCONUT did not outperform standard architectures in benchmarks)
- Steganography is so far proving quite hard for models to do (examples are here and here and here)
- For all of these reasons, models are very bad at evading CoT monitors, and the forward pass is also very weak computationally at any rate.
So I don't really worry about models trying to change their behavior in ways that negatively affect safety/sandbag tasks via steganography/one-forward pass reasoning to fool CoT monitors.
We shall see in 2026 and 2027 whether this continues to ...
2[deleted]
[This comment is no longer endorsed by its author]
1If everyone reads it, everyone survives?
3