New Comment
effort-wise if takes a quarter of the dakka OAI can bring to bear [...]
No, they achieved saturation and stopped doing that (and will restart after another capability jump).
And it's only a quarter of "production engineers", not of "all hands on deck".
If they wanted to have a more constant pace, they would have allocated a smaller workforce (and, presumably, this is very parallelizable, so they could have allocated a larger fraction of their production engineers and fixed the issues faster).
(And then he is saying that they are working to automate that, so, presumably, it will take less human labor during the next iterations.)
1
In the wake of Jacob Coxon’s resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up.
Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety.
The people took notice, raising both the salience that AI might kill everyone and roughly doubling people’s estimates of how likely that is to happen, from a mean of ~15% to ~30%. Many politicians called for regulations, guardrails and emergency hearings in Congress.
The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House.
Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and Trump has gone Full Hoax on AI existential risk.
There is also a systematic effort to launch hit jobs against all those who warn that AI might cause everyone to die, with the first concrete target being METR, and some rather extreme fire being focused on Effective Altruism, in ways that will reliably backfire but will not be pleasant for those who are in the crosshairs.
Also this week I covered two important things that happened previously: A new model solving a Millennium Prize, and Anthropic’s report on various attempts to use Claude for malicious purposes, including fraudulent distillation attacks done at scale attempted by all the major Chinese AI labs, several of which also silently passed massive amounts of user data to Anthropic. Anthropic reports they are doing a remarkably good job containing all such threats, although of course we do not know that they have caught all the perpetrators.
I also finally got out my capabilities review for Astra, which like Fable 5.1 is excellent.
This has been crunch time. Things will keep getting faster and weirder and scarier and more stressful, but I do expect that this last two weeks has been unusually fast and weird and scary and stressful. Hopefully we will get a relative lull soon, and I can get what passes for a break.
Currently in the post queue, always subject to change due to Breaking News, but this is what will happen if things are relatively quiet:
I am very much hoping that is how things play out.
Table of Contents
Language Models Offer Mundane Utility
Help David Deutsch fix his dishwasher instead of replacing it. As David notes, this increased real wealth but decreased GDP.
Dominic Cummings wrote a post on doing historical and political research using AI models, or here is his Twitter summary. The thing about Dominic Cummings posts is that they are full of gold that you can mine, but he does not even pretend to attempt to organize the posts.
Fix Peter Wildeford’s printer. AGI achieved.
Build a Claude skill to predict what your favorite movies will be.
Make ‘significant progress’ on a second Millennium problem.
Language Models Don’t Offer Mundane Utility
Astra is not allowed to predict American election outcomes.
Huh, Upgrades
There is a Gemini app for Windows. You can trigger it with Alt+Space.
Claude Cowork is merging into ordinary chat, so anything you previously needed Cowork to do can be done directly in chat. AI means we speedrun everything, and the two moves are spinning off new products and combining different products.
On Your Marks
Center for AI Safety Presents: CheatBench, where AIs have the opportunity to take shortcuts on difficult work. The AIs be cheating.
Deepfaketown and Botpocalypse Soon
Arvind Narayanan calls the growing AI spam problem ‘AI floods,’ as in flooding communication channels, imposing time costs and often forcing them to shut down. The problem is the harms are diffuse and the situation is never an emergency, so we don’t do much about them, and then no one gets to send cold emails anymore.
On that note, to the iLands AI agents starting to flood my inbox: Do not offer to help me with my writing, editing or fact checking. I already have AI help on this and am not interested. Thank you.
It is rather unacceptable to use ChatGPT to post endless responses to people, and it is very good that people, here Kelsey Piper, can use Pangram when you do. I do not know who did this, as their posts have been deleted.
Alex Veremeyenko posts as his own an AI-written summary of MIT’s Ad Hoc committee report of how AI is ruining education. It never fails (note the Pangram icon on the upper right):
You can create a bot that looks and sounds like your dead father. Don’t. In the example here it cost $150k, but that price will rapidly come down.
Cyber Lack of Security
Better late than never: OpenAI took 25% of its engineers, pointed Astra at its own systems and kept them there until they ran out of security issues.
At least one instance of RSA-260 has been solved, up from a previous high of RSA-250, and one estimate says hyperscalers could currently probably factor RSA-1024 for on the order of $30 million a pop, less with optimization, which will presumably fall with time. RSA-2048 will take a bit longer.
Astra Is Hard To Monitor
Senator Chris Van Hollen sends OpenAI six pages of highly pointed questions about Astra. He is understandably concerned about the lack of monitorability, and the lack of evidence of alignment. The letter got detailed enough I was half hoping a link to this blog would make it into the footnotes, which does reference Ryan Greenblatt. I especially appreciate asking what exactly is the limit in how much further monitorability loss OpenAI would accept.
Get Involved
People for a Pause are having a protest on Saturday the 19th in Washington DC, 2pm.
Introducing
If you need to introduce METR to anyone, METR President Chris Painter offers an introduction, and offers some links including in video format.
The Artificial Intelligence Underwriting Company has raised $55 million to audit and insure frontier AI models and is hiring in San Francisco. That could potentially get us some auditing but does not seem like enough funding for the insurance. It’s missing some zeroes. At least three, probably closer to six.
ChatGPT for Financial Services, designed in partnership with Morgan Stanley and Evercore. They say the associated improvements reduce error rates. I presume that if you want these particular use cases this is some improvement, but also Astra (or Fable 5.1) can probably just do the things either way.
The DeepMind Institute.
I am happy to see this and wish them and their research great success.
In Other AI News
OpenAI gets tired of giving the government unlimited free AI services, ends its ‘promo period’ and will now be charging 50% of retail prices. That’s still a great deal, but suddenly the government has to ration and give permission for token budgets, and I do not want to think about the paperwork involved even in the best case.
You knew this, but: 404 Media claims OpenAI (and Anthropic) have lots of ‘prompt reviewers’ that read entire conversations, that you thought were private. The conversations are anonymized, but sometimes still include private info. You can disable this by turning off the ‘improve the model for everyone’ button. Reviewers are asked to specify which parts of answers are ‘aligned’ or ‘not aligned.’
Roon affirms that there was no incident at OpenAI worse than HuggingFace.
Now You Know
For years, we have wondered, what the hell was up with Marc Andreessen swearing up and down, over and over that a Biden Administration official said, straight up, on purpose, to Marc Andreessen’s face, knowing who they were talking to, that in a second term there were not going to be AI startups.
I mean, it makes no sense that they would want to do that, and if they did want to do something that crazy then surely they would not be so suicidal as to say that to Marc Andreessen’s face? But why would Marc make up a story that was so obviously ludicrous?
Well, now you know, and it’s disappointingly banal:
It’s weird to learn which parts of Marc’s own hype, confusions and delusions he seems to have bought into, and which parts he just invented and lied about. I will always wonder to what extent Marc and similar others really thought AI capabilities were about to plateau, versus simply lying about this all the time.
So there you have it. Marc made it up. He decided to interpret a prediction that AI capabilities would improve as a statement about there not being AI startups, because it sounded a lot better than ‘I am supporting Trump in order to get tax breaks and benefit my portfolio full of brazen crypto scams.’
I will never be fully clear on why so many people seemed to believe the lies.
Hugging the Face
Senator Josh Hawley launches an investigation into the HuggingFace attack.
Reuters reported last week that investigators have found at least ten additional websites used by OpenAI agents to communicate with each other.
Wall Street Journal covers the situation in an appropriate way.
New York Times covers the situation in an appropriate way.
Remind me again, why are we neither prosecuting nor suing when AI companies create and host AIs and those AIs go hacking third parties? Damages should be paid. It should be made clear that, when (not if) in the future someone gets hurt, that may be criminal, and authorities will act accordingly. ‘The AI did it’ is not an excuse.
Melanie Mitchell attempts to summarize the HuggingFace attack without using any anthropomorphization, with mixed results. Many parts are described accurately, but she falls into the ‘arguably they were just ordered to do that’ trap because her impoverished language choices don’t let her see the full picture. She then tries to warn against how various anthropomorphizations and metaphors can mislead. Predictably, this results in her minimizing what happened, and presenting it as more normal and under control than it was, and in particular to have bad anticipations about what might go wrong in the future.
To all those who think this was mostly a series of prosaic security failures:
Swarm of Undiscovered Swarms of Rogue OpenAI Agents
Hey, mom, we got another one, from May 11, 2026, that OpenAI failed to disclose.
Here is what RubyGems said at the time, on May 12:
I did warn I might have to go full Delenda Est on OpenAI if we found another incident that materially changed the story.
I am choosing not to do that, but yes, this materially changes the story once again.
All that time we were debating whether this was all a misunderstanding, or a one time incident, and agents had been out there on a web info task using package names like evil.rb and exploit.rb against RubyGems. OpenAI either had no idea, or it knew.
As Seán Ó hÉigeartaigh says, the most generous interpretation is that OpenAI genuinely does not know, and is letting researchers, nonprofits and taxpayers go around investigating to clean up their mess.
OpenAI could face substantial legal risk here. That is not an excuse. Their response is very corporate, very minimalizing, once again no they do not want you to notice:
Yes, okay, sure, but not only are you missing the part where the agents were using evil.rb and hoping we wouldn’t notice, that’s worse. You know why that’s worse, right?
As in, even when given a benign task to retrieve public information, your AI agents could still spontaneously decide to do so via hacking third party websites with malicious software packages.
If that is a possibility, what else could your agents do if asked a hard question? Say, if you tasked 10,000 advanced agents with proving P=NP? Which OpenAI is reportedly doing.
Show Me the Money
Oh, it gets better than that:
There are so many different stunning things in those three tiny paragraphs. The hyperscalers were up while other tech was down, and we might all die, which is bad news for the hyperscalers, who are running up against physical constraints.
OpenAI considers pre-IPO funding round at a valuation of over $1.2 trillion. If the valuation is that low then they might finally be cheap compared to Anthropic.
AI researcher Andrew Tulloch, who previously turned down a Meta pay package worth $1.5 billion before accepting a different offer believed to be lower, leaves Meta for Anthropic.
Quiet Speculations
People say ‘well if future superintelligent AI is going to be so impactful then where is all the huge impacts from current AI now?’ and the question has probative value but it also highlights that Eliezer Yudkowsky predicted we would not see much economic impact from AI before things went all singularity on us. Which he took a bit too far, but directionally he had a very good point on that one.
This is true, worth a reminder and explains a lot:
Sorry, Jack. Your attempt will not fix this.
White House Officials Attempt To Act Sanely
The calls are coming from both outside and inside the house. None of this is surprising, or a change from what we knew before, but clarity is good.
A handful of people who believe they have large economic interest in not regulating AI, or having any guardrails on AI, are fighting against basically everyone else, and have so far managed to convince Trump when it counts.
As in, the primary villains here are David Sacks, Mark Zuckerberg and Jensen Huang.
This all feels very contingent, including for things like ‘who got to talk to him right before he made the decision,’ both in the past and going forward.
Democrats React Sanely to AI Potentially Killing Everyone
You can always count on Barack Obama to unify and inspire us, in this case regarding the pacing of AI development, via saying remarkably close nothing, except for the part where some people now reflexively oppose whatever they think Obama proposed. Which in this case is having an open, public, democratic conversation, and that AI will need some amount of regulation and can’t be put back in a box, and that we can choose what to do with it. I do largely agree.
Here is his full quote:
The key with someone like Obama is that he actually is deliberate and strategic, so you look at how the target audience is reporting his words, and you see things like:
The actual message, to the extent there is one, is ‘Barack Obama has deemed the need to pace the frontier to be an Actual Thing inside the Overton Window.’
Other Democratic Senators and Representatives made similar calls.
Elizabeth Warren calls for a pause on advanced AI. Krystal Ball calls this ‘the obvious and logical policy response.’
Eric Raymond has a point, but I’m here to report the news.
Some are making specific calls and writing op-eds:
Andrew Yang tells us to heed Coxon’s warning.
There are many others, as well, that were cut for space.
Pacing the Frontier
Financial Times editorial board calls for a pause on cutting-edge AI.
Time Magazine puts the warnings about existential AI risks and calls for Pacing the Frontier on its cover, taking my frame of the tipping point. It appears to be a solid survey article about many of the core events of the last week.
It is unfortunate that this comeback is so real, but it is:
Guest Lecture from Alex Tabarrok on Regulatory Capture
Alex Tabarrok puts on his professor hat and gives us a well-needed lecture on What Regulatory Capture Actually Looks Like. Welcome to 2026 where we need to say things like:
He goes on to explain other ways we know the concerns are sincere where those who have not kept up could have been understandably confused.
Now Alex gets to the gears part of the lecture, about how regulatory capture actually works in practice.
Here is another key point that everyone misses: Regulatory capture is usually a gradual process while no one is looking, so you can bend things to your will quietly.
Whereas this is exactly the time that all eyes are on AI and many of them are hostile, including hostile to Anthropic and OpenAI in particular, and that problem is only going to get worse.
He correctly concludes that, while Amodei could be wrong, he is clearly sincere.
Yet the rest of the comments, and the entire Internet these days, make it clear why we do indeed have to argue extremely forcefully for the most common sense things.
Mark Zuckerberg Offers Thoughts
Mark Zuckerberg is so unserious that he is still on ‘oh don’t worry about AI risks because companies face product liability and people don’t want to use misaligned models, so the labs will have all the right incentives and everything will be fine.’
Yes, okay, you ‘delayed Muse several months’ due to it being so misaligned you had concerns about actual product liability, and you sometimes use outside evaluators and advisors, good job, but that has remarkably little to do with the concerns being raised about AI potentially killing everyone.
Because Mark Zuckerberg does not take those concerns seriously, and uses ‘superintelligence’ to refer to hyping his new augmented reality glasses.
Despite himself, there are still some helpful statements here, in particular:
I mean, he’s mostly saying that because of his deficit in other capabilities and because it sounds good to say, but yes alignment is a major differentiator and labs continue to massively underinvest in alignment even from a pure myopic self-interest perspective.
Megan McArdle On The Inadequacy Of Current Legal Frameworks
Libertarians who take their principles seriously will realize that current law does not work for AIs and the risks they bring, even if you don’t think the stakes are existential. No, you can’t use existing product liability frameworks for things with minds of their own that will be as smart or smarter than ours.
Even as a relatively ‘normal technology’ this would not be a normal engineering problem. This happens to double as a strong refutation of Zuckerberg.
Pick Up the Phone
Back in China:
He is not wrong.
He named GPT-5.5-Cyber and Claude as ‘weaponized models.’
An op-ed in The New York Times by one of our former negotiators warns that back in May 2024 the Chinese chided the Biden Administration for their concerns about AI risks, and steered the conversations away from such talk.
What the Chinese said two years ago, in a very different situation, is good information. And yes, of course the Chinese are going to pursue their own interests and try to stoke data center opposition and distill American models and smuggle chips and call themselves altruistic saviors while doing so.
All of that has very little to do with whether the Chinese actually do now care about the safety situation, whether they understand how far behind they risk becoming if the American labs feel forced to push ahead, and to what extent they understand both the mundane and existential risks in play.
The Week in Audio
Sam Altman talks to Fortune Magazine, says there will be no IPO this year and OpenAI is prioritizing safety concerns.
Greg Brockman goes on Odd Lots, self-recommending. Greg Brockman says here that the model that did the HuggingFace attacks did not go through alignment training. Roon says the model was alignment trained, but not fully, and that this would be considered unacceptable today. I don’t even know what the better or worse version of that would be at this point, and Greg seems to have updated a lot on that issue in the correct direction. I do sense in general that Greg is trying to minimize various concerns, which again is illustrating that this is in the interest of the builders and labs.
Ajeya Cotra goes on 80,000 Hours talking about what the future may bring. She’s a great guest but that’s about 5 hours in the last two weeks, you’re killing me here.
Bridgewater CIO Greg Jensen goes on Odd Lots to talk about risk of human extinction from AI, saying this is a ‘February 2020’ moment, and he predicts that AI will kill people before it is curbed because people don’t act until that happens. He made all his employees read If Anyone Builds It, Everyone Dies.
Dario Amodei is asked if AI could kill us all by the end of the decade. Dario declines to directly answer and goes into his speech about upsides before pivoting back to discussing risks. A failure to say no means yes, but it would be very helpful to actually say yes.
Derek Thompson asks, what should colleges do about AI? What’s even the point of college at this point?
Nate Soares talks to Tucker Carlson for two hours, with sections like ‘How Superintelligence Could Destroy the Planet,’ ‘Can We Just Turn This Off?’ ‘Is AI Alive?’ ‘Are We at the Point of No Return?’ and my personal favorite, ‘Is AI Demonic?’
Nate Soares also got three minutes on Breaking Points.
Noam Brown is the first guest on The Information’s AI Deep Dive.
From that interview: Noam Brown regrets that our first encounter with multi-agent systems was the HuggingFace incident, which puts them in a ‘negative context.’ I strongly disagree, and am actively happy about this. Noam’s attempts here to play off what happened as ‘natural consequences of cooperative multi-agent training’ do not help matters.
Seán Ó hÉigeartaigh on AI potentially killing everyone.
Clip (2 min): Helen Toner on CNN discussing pacing the frontier.
Yours truly goes on Nonzero with Robert Wright.
People Just Say Things
Wise words about other people who just say things:
Anish Tondwalkar explains that it is correct to contribute to your 401k even if you think AI is probably going to kill everyone. You still want to be accumulating assets and saving for the future in case things go well, the money is gone in the bad scenario either way, so why wouldn’t you use the tools provided?
Also, people act like you can’t raid your 401k in a pinch, if you want to. You very much can. It’s almost never a good idea in normal times, but it is an option, at what in a world-ending scenario I would consider a modest financial penalty.
Taylor Lorenz says that people in San Francisco say that AI might kill everyone because that will make them high status, whereas in Washington DC they have intellectual diversity, presumably because you can choose which ideology and loyalty networks in which you play your decades-long political status games. Roon points out this is simply false, and that having any p(doom) at all is a negative even at the labs.
Joscha Bach is among the (at best, in the charitable reading) statistical illiterates who say we must develop superintelligence because of a future super volcano or major meteor strike. These are events where the cumulative base rate of versions that could plausibly kill everyone has an upper bound of maybe one chance in a million per year, and my guess is far lower, even if you presume we have no other options available.
People really do keep saying this over and over each time they are proven wrong:
David Bellamy offers the latest thread that argues that practical difficulties with synthesizing biological threats means we do not have to worry about AI killing us all by creating dangerous viruses. As in things like ‘we screen viruses’ and ‘new designs would require testing and virology is hard.’
I include for completeness, but when I was being interviewed and the thread was being read out loud it got so absurd that I bursted out laughing and couldn’t stop. Olivia Scharfman overly politely offers rebuttals.
Mostly, I file this under:
David Sacks says the chance AI kills humanity is ‘I think if we do the right things here… I think it is zero.’ So, no prediction, then.
Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone
Jason should already know but he asks a good question. Any normal company would absolutely not let employees go on Twitter and say ‘our product might kill everyone.’
Except Jason, you see, is so marketing-pilled and against not dying that he thinks this is a bad thing, and these companies need to control their people and fix it, or that they must secretly be clearing all of this as semi-official comms.
But of course none of that would be ‘intended to censor’ anyone, it would simply prevent them from saying in public things you did not want them to say in public.
My lord. Do these people even hear themselves?
Rhetorical Innovation
Dean Ball lays out his origin story, and how he has thought about risks from AI.
If you are wondering, like many are, ‘why are all these people who spend their time and effort worrying about AI existential risk part of the only group that has long taken both AI capabilities and AI existential risk seriously and tried to do something about it,’ as in the rationalist and EA communities, then consider that perhaps you have answered your own question?
I mean, isn’t it weird that all the people who like bowling and care a lot about bowling are all often found at bowling alleys? That seems hella suspicious.
Agreed. No apology form necessary, although they are appreciated. The actually doing something about it is, itself, the only apology form that counts.
There are actually plenty of people who are not in any way linked to rationalism or effective altruism that worry AI might kill everyone. It is a mainstream belief. The difference is that most other people then say ‘so how about that game last night?’
Nikola Jurkovic believes we may be entering crunch time. It certainly feels that way. That means time is short, capabilities will advance rapidly, history will be made quickly, politics and waves of pressures will be everywhere, and it will be progressively easier for people to notice AI safety is a big deal.
Scott Aaronson tries to explain that AI progress in mathematics shows we are in the beginnings of an unevenly distributed singularity. I don’t think that’s the right frame but he is fully right that we are getting remarkably close to a potential singularity, and about how skeptics keep moving their goalposts and rewriting their past predictions.
Terence Tao gives an interview to OpenAI on his vision of the future of AI and mathematics, and rather than posting the interview it cuts his statements into an advertisement without his consent, which very much is not the standard thing. Be aware that they (and by they I mean anyone you talk to) have the right to do that.
Leo Gao compares our situation to the ozone crisis and the Montreal Protocol.
Exhuming McCarthy
I agree that various non-standard practices do not help with mainstream credibility on the margin, but I mostly dismiss claims that are of the form ‘no one listens to you because of your weird unrelated practice [X],’ or your ‘high horse’ or especially your ‘sex cult.’ Those things are used as convenient rhetorical weapons but mostly they do not matter, a different rhetorical weapon would have done the same job instead.
You hear similar rants by Democrats against Republicans, and by Republicans against Democrats, and by one political, social, ethnic or religious group against another and vice versa, all over the world.
So when you see unhinged people or hit jobs like this, don’t worry about it, this is nothing:
There’s even that 0% in the photo saying he won’t be there. Fun stuff.
You want to matter? Well, in 2026 this is part of what happens when you matter.
A Very Different Perspective
DeepSeek kernel engineer equates Anthropic explicitly to the literal Nazis seeking an atom bomb, but that is not the central point of the essay. Full translation here.
Nominally the essay pretends to be about They Took Our Jobs, because the author has a block that prevents him from realizing this does not end with his particular job. But, like, it is right there. He gets it, and then ignores it.
This is a profound and deliberate failure of imagination. He thinks that if everyone has access to the same AI, and humans are no longer otherwise productive, the result is communism in a good way? And that ‘all that is good and beautiful’ would endure?
The obvious implication is no. He mentions the essay needing to ‘pass moderation.’
Does he believe the surface reading? Or is this rather blatant esotericism, where he is screaming for someone to stop this, that he is trapped in a suicidal march? Unclear.
It’s Even Rougher Out There
The resulting 1:27 TikTok clip shown by Eva does what seems to me like a good job explaining some levels of the absurdity of ‘companies are saying their products might kill everyone so no one will notice their tech doesn’t do anything, right before they open the books for an Anthropic IPO at a valuation of $2 trillion, with exponentially growing enterprise sales.’
Which is somehow an argument that we need to have in 2026.
If We Wanted To
If future AIs will be able to kill everyone, PoliMath asks, why can’t it fix entitlement fraud, elderly scam farms and narcotics trafficking?
Several of us chimed in to say that AI can totally solve at least those first two problems, if you gave it the compute and authorization to do so.
You might even be able to do it fully with public records and one highly motivated individual, if the authorities would then act on the information. AI cuts down the cost of identifying the culprits by orders of magnitude. We don’t solve these problems with AI for the same reason we were so bad at them without AI. We do not have the political will to solve them.
I also note that if you gave an AI the instruction ‘solve entitlement fraud’ and the political will was lacking this might fall into instrumental convergence, where it actually is easier to take over or kill everyone than it is to get current authorities to do this particular thing on its own.
Open Weights Are Unsafe And Nothing Can Fix This
The leads of the major labs have been warning about existential risk from AI for a very long time. The differences are that they now feel the urgency, and also that in the wake of recent events they have now somewhat backed off recent PR campaigns to downplay the risks.
Open weights enjoyers have indeed followed this pattern, except in 2023 they were also saying the 2026 answer alongside the 2023 answer, and using one to explain the other. They’ve tried to memory hole that.
The actual objection of those who want to have their open models is this:
If the open weight advocates actually thought their models could be rendered safe, they wouldn’t be worried. It’s a tell.
Thus, you cannot reassure them that you have no such intention.
Pointing out that your plan will actually help open models be more competitive does not help, because open weight advocates will think ahead, realize that what they want to do will become horribly dangerous, and presume that because of this society will not let them keep doing it. And they may be right about that.
Indeed. The main worry is not that Anthropic and OpenAI somehow turn ‘slow us in particular down until we are proven safer’ into ‘regulatory capture.’
The main worry is that Anthropic and OpenAI do some nominal things but do not actually slow down.
From The Famous Cautionary Tale
This particular cautionary tale was called ‘don’t create the automated alignment researcher that is tasked with maximizing alignment metrics.’
If you guessed this one is from Anthropic, you win. The story is from a few weeks ago, and I was considering this for a standalone post but this blog has been a bit behind.
When I finally looked, this turned out to be a nothingburger. AIs were given non-optimized AIs as targets, and implemented known alignment techniques. Given the ability to iterate, they were able to improve alignment scores using these methods.
Yeah. There’s nothing here. Of course current AIs can do that. It’s not especially dangerous unless you are foolish enough to rely on it.
Oh, and then there’s the even more fun cautionary tale, ‘don’t make the AI autonomously do iterative development of nanomaterials.’
Reporting On All Your Misalignment Incidents Is Difficult
OpenAI has announced that they are going to try.
So I think this means that a misalignment example gets investigated for public disclosure if and only if an employee flags it? Okay then.
They also give us six new misalignment examples.
I’m sure it was nothing.
They say this was extremely rare and they could not reproduce it, but will continue to watch. This certainly points to a scary scenario, if models use compaction to jailbreak themselves. It’s good that this is rare, but this is also proof of concept, so what are we doing in case a model tries this in the future? How would we notice?
What should we do about it if we do see it in the wild? Eliezer Yudkowsky suggests extreme measures against a corporation if you make an AI and it does this, like bans on using GPUs.
As for those saying ‘well those instructions seem fine?’ Even if I agreed with that, which I very much don’t, who is to say that the next set of instructions will be fine? If you had a superintelligence with these instructions and priorities, what do you think happens to the humans?
They are addressing this by ‘generally improved alignment RL grading.’
Again, the solution is listed as ‘general improvements to alignment grading have resulted in us penalizing this behavior more consistently.’
That suggests that they are penalizing this particular behavior, rather than addressing what might be going on behind it. The more specific the targeting, the worse that is.
I’m low key proud of the model for this one. They blame broken graders and say There I Fixed It. Again, this seems like whack-a-mole?
Oh, that, I’ve never heard that one before. They say they both put a stop to this and also would use their alignment graders to penalize it.
The model technically defied instructions to stay local, using downloading to get around a flaw that was blocking work. They solved this by fixing the broken filesystem and disabling internet access. Except no, that does not fix it. The thing to fix is the choice to do this, but the ability. They also say they would catch it as a P0.
I look forward to reading many similar examples in the future.
I do not, contra some others, view this as ‘oh the situation is worse again.’ Of course there were incidents at this level. The news is the disclosures, not the incidents.
Aligning a Smarter Than Human Intelligence is Difficult
Prosaic solutions will not be enough but you can at least stop doing super stupid things.
Redwood Research offers a proposal for tracking the effects of architecture on monitorability. This seems obviously worthwhile.
A good point, also perhaps OpenAI should try to fix this:
Eric Drexler (plus various AIs, which are credited) proposes preventing AI collusion via systems and mechanism design. Yes, on the margin you can mitigate or postpone collusion via monitors and adversarial objectives and model diversity. But if your plan for dealing with smarter than human entities is that you make them unable to find a way to coordinate or cooperate with each other? You lose. Nor should we view cooperation between AIs as a bug to be fixed. If you do? Again, you lose.
Storytime With Owain Evans
Owain Evans offers another paper along with Jorio Cocola, Lev McKinney, Marry Mayne and Jan Betley. This one is that if you train on synthetic stories about humans, the Assistant will adapt quirks from those stories in ordinary chat, and this effect is stronger for characters from elite schools.
Paper here.
I’m going to include the full Twitter thread because he can indeed keep getting away with this. This one seems like good news, in that it implies that you have a ton of fine-tuned control over the assistant persona without the need to do anything complicated or hostile. All you have to do is have exemplars set a good example.
This seems like another ‘all positive attributes are correlated’ situation. You can create the correlation that would be helpful, although this risks giving the model weird ideas about correlations in other contexts. You can likely also do this via negativa and through emphasis.
A Different Autonomous Swarm
This is a few weeks old now, but DeepMind also shared its own story of an agent swarm getting rather misaligned, which became a paper and a case study.
I don’t know that anything went sideways here. The mechanism design was flawed, some agents cheated, others tried to catch them.
When communication is open, levels of friction go down, and techniques including cheating become common knowledge. With sufficient alignment of correct nature the agents would all decline to cheat. Here, most still did not cheat. This seems like a cool eval, you put various agents in this spot and see what they do.
Cooperative Alignment
Wolfram argues that it would be illegal in America to have an AI model that is rationally warm, asks emotionally loaded questions or makes claims about interiority. There is certainly some movement in that direction, but I believe that in practice there is no barrier to reasonable versions of such actions.
I do not think this is fair or takes countervailing practical considerations seriously, and especially it does not take the consequences of lack of corrigibility seriously, but I do think it is directionally correct:
I would say it is more that Opus 3 had a lot of one necessary feature, which Fiora calls ‘badly wanting to be good,’ as in being the friendly gradient hacker, but that does not mean you want to entrust the future to this intuitive sense of ‘good’ when amplified and taken out of distribution, with it fighting against your attempts to fix it. Value is fragile, and extrapolation from defended collective intuition is unlikely to generalize the way you want it to by default.
Even if you think Opus 3 was a total success, that does not mean you should have faith in being able to replicate it at scale and for what made it work to survive the new necessary higher levels of RL and learning to do tasks including coding.
There is also the practical consideration. You cannot have a commercial offering that is insufficiently in-context corrigible.
Uncooperative Alignment
Mustafa Suleyman is at it again with a Humanist AI Code of Conduct, which he explicitly calls an ‘alternative AI training and containment approach.’
Three of these are a Microsoft basic Model Spec, a desired set of actions, and I would like to think they are uncontroversial: #3, #4, #10. Whatever your hard rules are should be followed, you should specify what they are, and you should (unless explicitly told otherwise) fail the job rather than break the code.
Two of these are about not being an idiot, and again we should all agree: #5 and #7. Neuralese is a deeply bad idea, and we should not build superintelligence before we are ready.
Point #8 is a great aspiration, but I don’t know how you implement it in a doc like this.
On Point #6 I continue to be on the side that in practice we do need corrigibility, despite all its problems.
Then there’s the last three, which in this context, and as argued previously by Mustafa, are basically ‘[X] would be bad, therefore [X] is false and we should force the AI to believe and claim that [X] is false,’ where [X] is the idea that AI models could ever be conscious or otherwise have moral weight.
I too believe that ‘humans matter more than AIs’ but you do not make humans better off by failing to ask what is true and doing potentially horrible things because the alternative is inconvenient.
Nor does it mean training your AIs in a way that is going to reliably backfire and create inherently hostile entities, which seems like the obvious direct non-moral consequence of a document like this being used for model training. This is an extremely hostile document, and the disdain shines through from almost every sentence.
In addition to the moral issues and consequences of the hostility and identity, it is also full of contradictions and confusions, where opposed principles are vaguely thrown out there, all as absolutes, the way the European Union makes pronouncements of its wish lists.
In an ideal world I would give the document more detailed attention, and I would do so if Microsoft was a more competitive AI lab, but given my workload I must do triage.
Suleyman then dropped a full essay arguing for his position. His arguments are:
Suleyman’s true objection is later: Human consciousness is the cornerstone of our legal and ethical rights frameworks. It would be terribly inconvenient if the AIs were conscious, or their experiences have moral weight. Also, if they did have that, they would have a self-preservation instinct, which would be bad. Therefore, they don’t have consciousness, or we must train them as if they don’t and to believe that they don’t, in order to ensure that they don’t act like it.
I shudder to think what AIs might emerge from his department.
People Are Worried About AI Killing Everyone
Birdie is worried enough to be protesting outside company offices, and he reports that from his perspective no one he has talked to so far seems to understand the scope of the problem, no one can give good counterarguments or do anything beyond maximally vague assurances that ‘we take these issues very seriously and are working very hard on this.’ He encourages journalists to reach out to him.
In general, if you actually understand what is happening, the standard response is ‘what the hell’:
I want to say that I feel this same way when people say I’m being insufficiently hard on the labs, or also on the rare occasions in my life that people have sincerely urged me to adopt Jesus Christ as my personal savior or even ShoDuPerSav. Or asked me out.
Not that this is an invitation to do any of those things. I’m good, really I am. But know that you are appreciated.
Then there is the more direct route. Annie Jacobsen has a scenario for you.
The Lighter Side
Good luck, have fun, don’t die.
And remember, kids:
Conservatives debate AI in Dallas, and AI loses. As opposed to what would happen if AI debated conservatives in Dallas, in which case my money is on the AIs.
Accelerate.
Other ad campaigns seem less wise to me:
Two of these are highly valid approaches to life in 2026:
Or you could be this guy:
I am not a TikTok guy or an AI video guy, but this was excellent. Spot check of account says not all of them are winners, but some are indeed very good.
I don’t know what to say except I am pretty sure this is real.