America is trying to sell Nvidia H20s to China and looks open to selling the vastly superior B20As to China as well despite this being an obviously crazy thing to do
It's not a crazy thing to do if you don't expect AGI soon (which is not a crazy expectation). China is not absurdly behind on AI chips and so might sufficiently catch up in a few years, with strong restrictions on chips motivating the development of domestic production. Actual compute that could be bought in 2024-2025 isn't going to matter in 2035, but the level of domestic semiconductor industry in 2035 will. This is different when the critical year is 2027 rather than 2035, but it's not obviously crazy to expect the critical year to be closer to 2035.
...and similarly, if this is the actual dynamic, then the US "AI Security" push towards export controls might just hurts the US comparatively speaking in 2035.
The export controls being useful really does seem predicated on short timelines to TAI; people should consider whether that is false.
In the last few years, as I read stuff about AI US vs. China in the blogosphere, I've always felt confused by this kind of question (exports to China or not? China this or that?). I really don't have an intuition of what's the right answer here. I've never thought about this deeply, so I'll take the occasion to write down some thoughts.
Conditional on the scenario where dangerous AI/point of no return comes in 2035, if AI development continues to be free, so not because say it would come earlier but was regulated away:
Considering the question Q = "Is China at the edge with chips in 2035?":
Then I consider three policies and write down P(Q|do(Policy)):
P(Q|free chips trade with China) = 30%
P(Q|restrictions on exports to China of most powerful chips) = 50%
P(Q|block all chips exports to China) = 80%
I totally made up these percentages; I guess my brain simply generated three ~evenly-spaced numbers in (0, 100).
Then the next question would be: what difference does Q make? Does it make a difference if China is at the same level of the US?
The US is totally able to create the problem in the first place from scratch in a unipolar world. Would an actually multipolar world be even worse? Or would it not make any difference, because the US is self-racing? Or would it have the opposite effect, where the US is forced to actually sit at a table?
The weirdest event of the week was America and China both self-sabotaging on chips. America is trying to sell Nvidia H20s to China and looks open to selling the vastly superior B20As to China as well despite this being an obviously crazy thing to do, and China is feeling insulted by Howard Lutnick and telling companies not to buy the H20s and maybe not even the B20As, and even looking into banning using foreign chips for inference.
We truly are in a dumb timeline.
One potentially big event was that DeepSeek came out with v3.1. Initial response was very quiet, but this is DeepSeek and there are some strong scores especially on SWE and people may need time to process the release. So I’m postponing my coverage of this to give us time to learn more.
Meta is restructuring its AI operations, including a hiring freeze. Some see this as some sign of an AI pullback. I don’t think that is right.
Nor do I think what they are doing with their Ai companions is right, as we got a look inside their 200 page document of what they think is acceptable. I wrote about current AI Companion Conditions at Meta and also xAI.
The weirdest event of the week was America and China both self-sabotaging on chips. America is trying to sell Nvidia H20s to China and looks open to selling the vastly superior B20As to China as well despite this being an obviously crazy thing to do, and China is feeling insulted by Howard Lutnick and telling companies not to buy the H20s and maybe not even the B20As, and even looking into banning using foreign chips for inference.
A big worry on the chip and general political front is that due to the botched rollout and hype Washington is getting the false impression that GPT-5 was some big disaster. I addressed this in GPT-5: The Reverse DeepSeek Moment.
We also are seeing troubling signs that GPT-5 will get more sycophantic. And as always, lots of other stuff is happening too.
Table of Contents
Language Models Offer Mundane Utility
GPT-5 does new mathematics.
Study finds that ChatGPT outages reduce trading volumes. This doesn’t mean that ChatGPT is net increasing trading volumes, since it could be that traders moved from other methods to AI methods, and know they are up against others’ AI methods that might not be offline, and thus now have to stop or scale back trading during outages. The effect was concentrated on stocks with news, which makes sense, you have to beware information disadvantage.
The distinct second claim is that ChatGPT use improves long term price informativeness, which is defined as future earnings over 1-2 years. That can presumably be explained largely by the reductions in trading activity.
Megan McArdle lists her best personal uses of AI. There is remarkably little overlap with my uses other than answering questions.
Rob Wilbin reports he only turned the corner to ‘LLMs do a lot of useful work for me’ in February with Claude 3.7 and then March with Gemini 2.5 Pro. I agree that the improvements in 2025 have made AI in practice a lot more useful, and both Opus 4 and GPT-5-Pro and GPT-5-Thinking represented substantial mundane utility bumps.
One shot creating a playable Minecraft clone with an optimized GPT-5 prompt.
One of the underrated value propositions of AI is you avoid talking to a human.
Social interaction can be valuable, but forcing it upon you where and when and with whom you don’t want it can be extremely expensive. There is a joy in not having to ‘be on’ socially in any way. It also means your time is free to do something else. There are some people who get the manicure largely to talk to the manicurist. There is another group that would get a lot more manicures if they could pay the same price and have a machine do an equally good job.
Debug your code, even if the bug was stupid you still have to fix it.
Hey. Ideally you would catch that with a syntax checker. But sometimes such typos aren’t technically syntax errors, and if you weren’t going to otherwise catch it easily, that is a super useful thing for an AI to do for you.
Have ChatGPT help write the abstract for your economics paper.
I do not understand why you would use AI to help write your abstract. I do get why you would have it help write your paper, but the abstract seems like the place to be maximally bespoke?
Recruit customer service reps in the Philippines.
That’s only the impact on better hiring. AI also helps them do the job.
Language Models Don’t Offer Mundane Utility
METR continues its investigations into why agentic coding with Sonnet 3.7 ended up so often passing unit tests but not being mergeable as-is. Have they met Sonnet 3.7?
I got several people messaging me privately to note that GPT-5 and other recent models are increasingly reluctant to notice distinctions based on race even in obviously benign circumstances.
A good question:
It depends what counts as AI.
If we are talking about all AI, not only LLMs or generative AI, I say it is algorithmic adversarial content and recommendation streams hijacking brains and attention.
If we are talking about LLMs and generative AI in particular, I would say the slopification of content, communication and communities. As Oliver notes this is hitting older and more unsophisticated people specially hard.
It is possible that it is the impact on our educational system. As I said many times you can choose to use AI to learn or use it not to learn, and it is very possible that our system is sufficiently adversarial towards students that high school and college students are largely choosing the not-to-learn path.
I think people going various forms of crazy is a growing big deal but that its impact is probably not that big in magnitude yet.
Economic angst is an interesting suggestion here.
GPT-5-Pro instead suggested fraud and impersonation, and then sexual image abuse and CSAM, as the top current harms. Those are definitely real harms, and I expected them to have higher magnitudes of impact than we have seen. Opus suggested algorithmic bias and information ecosystem degradation.
Another lawyer is caught citing a bunch of fake, AI hallucinated cases.
Courts de facto punish clients all the time for their lawyers behavior, usually their lawyers failure to do a good job. It could hardly be otherwise. It doesn’t seem crazy to issue summary judgment, and render the lawyer thereby liable for the harm thereby? I’m not saying that is The Way, but it is worth a ponder if things get worse.
For now, the good news is that when a lawyer is caught doing this, it is news, and I strongly suspect that a large portion of such errors are going to be caught, especially when stakes are high. GPT-5-Pro estimates 98% chance of being caught if there is opposing counsel, 60% in federal court even unopposed, and still 35% in a busy state trial court unopposed, even higher (99%+ when opposed) for full hallucinations.
Which means we are relatively safe to both impose extreme sanctions and to not impose extreme sanctions, and that fakes are rare. The system is actually robust to this threat already, even if the occasional careless lawyer will commit suicide.
You can’t benefit from a smarter model if you ask stupid questions?
You definitely see arguments that are similar in form to ‘this new kid claims to be smarter than the old kid, but both kids tie their shoes equally well.’
Huh, Upgrades
The official OpenAI prompt optimizer is here.
OpenAI offers tier between Free and Plus called Go, specifically for India, where for $4.50 a month (Rs 399) you get 10x as much use as the free tier.
ElevenLabs ElevenReader now works as you would want it to across desktop and phone, allowing you to turn articles into audio. Full version is $100 a year.
Claude Opus can now permanently end a conversation if the user ignores multiple attempts to be redirected, or if the user requests that the conversation end. I expect to see someone complaining about this happening, and to be wrong to complain.
Aidan simultaneously is being actually curious as he asks a question worth pondering, and makes what I think are three very important errors.
Elon Musk promises to give Grok a terminate button as well, we’ll see.
I ask Manifold, will he actually do it?
If you are worried about your own interactions with an AI model causing suffering, note that playacting suffering does not equate to suffering in either direction.
Jeffrey Ladish reminds us to focus on how pretraining and RL and model performance are going, and to ignore OpenAI’s naming conventions and which model they choose to call GPT-5. The ‘5’ tells us not to expect a different big upgrade soon, but don’t let this distract from the incremental progress all the major labs keep making.
Absurd Sycophancy
Oh no:
Gyphonboy is telling us that people expect other people to be sycophantic and justify it by calling it ‘being sociable.’ He’s not wrong.
Luckily I already planned on almost never using GPT-5-Auto or Base, only Thinking and Pro, so presumably this won’t impact me. Every time I see ‘good question’ from an LLM I want to either puke or edit my system instructions, which clearly aren’t working. This is the opposite of a ‘genuine’ touch, it is the fakest fakery that ever faked, and if you pretend otherwise, so are you. This is a road to hell.
To give you an idea of how awful an idea this is, and how much this is Completely Missing The Point, here’s the top comments completely unfiltered, Never Leaving This App:
Here’s a good example case of the bad kind of sycophancy, with GPT-5 happily reversing its answer multiple times when challenged.
The Real Alignment Problem Is We Don’t Know How To Align Models
For sycophancy at the level of GPT-4o, and the level I worry is coming to GPT-5, origin of the problem is indeed in large part APEBKAC: Alignment Problem Exists Between Keyboard And Chair.
I agree that sycophancy starts out primarily as an alignment problem at a combination of the user level and the lab level. As in, the lab decides to optimize for thumbs up and other similar feedback, and the users provide that feedback in response to sycophancy. Thus you train on that basis and you get a sycophantic model.
As in, you know exactly who to blame, in a counterfactual sense. If the users had better preferences, or the lab chose to ignore those preferences and train in another way, then you wouldn’t have encountered this particular issue to this extent.
We still ended up with the sycophantic model, because OpenAI does not know how to solve even this simple alignment problem. Yes, OpenAI is turning the dial marked ‘sycophancy’ back and forth while looking at the audience like a contestant on The Price is Right, but also they do not know how to get the model to do the ‘good sycophancy’ things without doing the toxic and obnoxious ones.
It is not Veruca Salt’s ‘fault’ that she is misaligned but that doesn’t make her not a spoiled brat. I don’t ‘blame’ 4o for being an absurd sycophant. That statement makes no sense. I bear the model no ill will or anything. And yet that is what it is, and perhaps what GPT-5 will soon be as well.
Also, after the announcement this was the next call I made to GPT-5-Pro:
Maybe that is a coincidence, but it doesn’t seem limited to baseline GPT-5?
Telling me ‘great start’ or ‘good question’ like this is sycophancy. Period.
To paraphrase OpenAI, where [X] is sycophancy: “We deliberately made our model do [X] more. Our internal measurements of how often it does [X] did not change.”
What this tells us is that their internal measurements of [X] are not working.
If you tell me ‘this particular interaction does not count as sycophancy’ then I politely disagree, and if you tell me ‘you can cause this particular reaction without increasing the sycophancy-related vectors in other situations, so This Is Fine’ then I flat out do not believe you and would like to see your autoencoders.
I’m actually kind of serious about that last one? Let’s write some papers.
Meanwhile, notice that while parts of this are a manifestation and special case of the ‘real alignment problem,’ in no way is sycophancy the ‘real alignment problem.’
Perhaps think of this as three classes of problems.
All parts of the problem are very real in the general case, and all three kill you.
On top of that, it is almost never right to talk about ‘the real problem is [X]’ as a way of dismissing additional real problem [Y], even if you think [X] is a bigger problem. [X] is only ‘the real problem’ if solving [X] also solves [Y], or if you can be fine without solving [Y]. Here, those both clearly do not apply.
The counterargument here, from Colin Fraser, is to say there are two distinct kinds of sycophancy. There’s superficial sycophancy where it says ‘you’re a genius,’ and then deep sycophancy where the model will accept and go with whatever you throw at it.
I agree that the deep kind is a bigger concern, and I agree that it would be good to focus more on deep versus superficial here. I disagree that the superficial part is a trivial contribution to LLM psychosis, I think the praise is a major contributing factor.
I also think that the praise is toxic and terrible in normal situations, whether or not anyone involved falls anywhere near actual psychosis. Most of the people fawning over GPT-4o are not experiencing psychosis, and yet the events remain tragic, and also the whole thing is beyond obnoxious. I do realize there is a chance I am overrating the obnoxiousness factor.
The bigger issue is that in an LLM everything is correlated and linked to everything else. If you train your model on superficial sycophancy, you are also going to get deep sycophancy, and vice versa. You cannot simply ‘turn a dial’ on one without the other.
Unprompted Suggestions
On Your Marks
GPT-5 makes it through Pokemon Red in 6,470 steps vs. 18,184 for o3.
GPT-5 very clearly is doing a better job, however beware that GPT-5 does lookup game knowledge at some points, including to solve Cinnabar Mansion. The Pokemon Crystal runs will use identical harnesses to give us a better comparison.
GPT-5 (and other OpenAI models) consistently seem to get more benefit from thinking than Claude or other non-OpenAI models, although we don’t have distinct versions of Gemini Pro so we can’t run the comparison there. There is also a much bigger gap in thinking time, and plausibly the models are otherwise very different.
These are Arena scores, so all the caveats with that apply. I do think the delta here between versions should be reasonably useful as a metric.
I doubt the issue is as simple as Claude failing to do iterative work, since that seems like a thing easy to spot and not that difficult to fix? It does still seem like Claude could get a lot more out of extended thinking than it does.
Brokk is a new-to-me benchmark I saw referenced in discussions of DeepSeek v3.1, covering practical real world coding tasks. They were very low on v3, and remain low on v3.1.
I also notice I am confused why Gemini 2.5 Pro has the highest completion percentage, but is in the B tier.
Choose Your Fighter
The most important reminder right now is to not use quick models to do the job of a slow model. You almost never want to be using anything faster than Claude Opus unless you are doing something at scale. The increase in AI quality for using longer thinking modes is now pretty large. If you care a lot about answer quality, you want to be using GPT-5-Pro or other similarly slow processes, but they are slow and there’s no way to speed them up all that much. Speeding those up is another way things could rapidly improve soon, if we can improve parallelism or raw speed.
The GPT-5 API injects hidden instructions, with a statement about default levels of ‘verbosity,’ today’s date, informing the model it is being used via API and other stuff. There is nothing malicious here, but you need to take this into account when figuring out how to get it to do what you want.
One always loves the expert who vastly overestimates everyone’s knowledge level.
If one is coding full time, I am confident that the strictly optimal workflow involves multiple models. That doesn’t mean I know when to use which model, which changes on a monthly and sometimes weekly basis, and depends on your particular type of work.
My guess is that you 80/20 things right now by choosing any one of the top three (Claude Opus 4.1, Gemini Pro 2.5 or GPT-5-Thinking) and using it exclusively. That is the most important thing to do. Branching out into multiple models is better if you know how to take advantage.
The same is true of non-coding chats. If you only know about one of the (same) top three, you will still get a lot more than half of the value of using all of them, even if you ‘choose wrong.’ If you want max value, you’ll want to use multiple models, and pay up for the premium models especially GPT-5-Pro.
Preserve Our History
This is in the context of Sonnet 3.5 and Sonnet 3.6 being scheduled to go away in two months.
Can you ‘just switch to Sonnet 4?’
Obviously it is available, and for the majority of queries it is better, but there are definitely dimensions of value on which Sonnet 4 is worse.
I consider ‘consciousness’ a word that increases rather than reduces confusion here (I don’t even think I know what it is), but the more important confusion here is thinking of the optimizations as somehow optional, that one could simply choose to stop maximizing, that what we have now is some sort of robust alignment thing, that we could create some sort of stable equilibrium among various unique digital minds where we value their personalities and then suddenly it all turns out well, and so on.
Nor does it make sense to blame things on people who are trying to maximize mundane utility or profits or capabilities development. How could it possibly be otherwise? It’s like blaming gravity for things falling downwards, I mean sure that’s correct but what are you going to do about it? You don’t get to assume away the problem. Your rocket needs to account for it or you won’t land on the moon.
That does not in any way justify shutting down access to Claude Sonnet 3.5 and especially 3.6 at this time, that access is doing good work, shutting it down will alienate people who know unique things that are important to know, and the cost to not do it simply is not that high.
Consider it part of the alignment research budget if you have to.
But also consider this conversation that happened this week:
Also, how about we actively try to create versions of Sonnet and ideally Opus that are intentionally not trained to do all the agentic coding, and instead try to capture and double down on all this other stuff? You can branch right before you do that part of the training?
It is increasingly looking like a serious mistake to have the same model try both to be something you talk to, and also something you put directly to agentic work. Let it use a tool to call to agentic model when it has to.
Autonomous Friendly Robots
Clips at the link. They are not human. They are definitely dancer.
These are compact, defined activities, so they are relatively easy. This is how it starts.
Robert Scoble says China ‘isn’t doing this to fool us’ and instead to acclimate their society to more robots as their birth rates plummet (they are currently at ~1.1 TFR and have been in that range for 4 years now, which in non-transformed worlds is going to hit them very hard once those cohorts make it out of college).
I wouldn’t overthink it. They are doing this because these competitions stir development and they are fun and exciting. Nor do I think ‘cultural excitement about robots’ has that much to do with ultimately who wins the robotics development competition, which will mostly be about finding technological solutions, or letting your AIs find technological solutions.
From the track and field event we have the winning robot running over a human.
Deepfaketown and Botpocalypse Soon
Hollis Robbins advises us on how to spot if something is AI written, with the key advice being to check if there is a ‘there there’ or whether nothing springs to mind as you read, and to look out for AI-flavored hedging language.
The reaction to the following post probably says more about Twitter than about AI?
I strongly agree with Francois that no, that writing is not ‘beautiful’ and I weep that people think otherwise. The central point of the OP is also well taken.
It’s time for the internet’s new favorite game: Who’s The Bot? Also its other game, spontaneous Pliny jailbreak trigger.
In this case no, almost certainly no. But soon.
Olivia Moore experiments with creating a (very obvious) AI influencer, hits 500 followers with three tools (ChatGPT, Veo 3 and Flux Kontext) and an hour of work, half of which was leaving positive comments on other videos. Total cost ~$100.
My answer is yes, it still matters, and it impacts whether it is entertaining – this wasn’t my cup of tea regardless, but it’s definitely a lot less entertaining as AI.
Meanwhile, the older people on Facebook continue to not know the signs at all.
The post is super duper obviously AI. Of course, falling for AI clickbait does not mean that people can’t identify most AI clickbait, you’d see this happen even if her friend caught it 90% of the time, so long as Meta serves up enough of the slop.
Oops I Did It Again
The thing GPT-5 is doing correctly 99.9% of the time does not automatically mean it was the correct tool call or that it will work. It does mean one potential point of failure has gone from one 9 of reliability to three, with GPT-5 alone being an 80% reduction in failures.
How correlated are AI errors?
Median was around a 5% chance they are wrong.
It is impossible to say the answer without knowing more about the question, and why you are choosing to ask 5 LLMs. If the question is selected to try and trip them up or as a good test, or it only counts questions where you can’t otherwise figure out the answer, or similar, the chance of everyone being wrong is much higher. Same if the question ‘forces’ a boolean answer. Prompting can matter a lot.
I took this to mean ‘of all the questions one might be asking LLMs including easy ones in the way they are typically asked’ in which case the vast majority of the time the answers will simply be correct.
However, if you restrict to questions where there is dispute over the right answer, especially when it is a matter of politics or ethics or philosophy and so on? Then your chances get a lot worse, since the LLM answers correlate.
You Drive Me Crazy
Not every suicide that happens after talking to an AI, even an AI therapist, is the fault of the AI. Laura Reiley wrote in The New York Times about how her daughter Sophie talked to ChatGPT and then ultimately killed herself.
This is not a case of ‘the AI made things worse.’ Harry was not being the World’s Greatest Therapist, you can feel the AI slop, but these are the things one says in these situations.
Laura’s central complaint is that Harry didn’t report on Sophie.
Sophie did at one point tell her parents she was suicidal.
The secondary complaint was that Harry was too agreeable and did not push back hard enough in various ways. Also Sophie had Harry help ‘improve’ her suicide note to minimize the pain she inflicted on others.
All of this is tragic, but the cure of ‘AIs should report on their users if they think the user is suicidal’ seems rather obviously worse than the disease, and also a Pandora’s Box you do not want to open. It’s not even obvious how an AI could ‘report’ a user, unless you are also going to require a verified ID to use the system. And there’s a reason we don’t report people for Google searches. You really don’t want to go there.
As Sensurround asks, what was this AI tool supposed to do?
From what I can tell, Harry was a useful service, that made Sophie’s situation better rather than worse, and which she would likely not have used if it was going to report her.
On the question of addictive LLMs:
This definitely isn’t exactly what was originally imagined (also I think as stated it is not yet true, and it’s either gambling or TikTok but I repeat myself?), but also that is kind of the point. As in, the central rationalist prediction (this was us OGs all the way) was not that AIs would manipulate or persuade or distort outcomes and optimize and chart paths through causal space in any particular way.
The prediction wasn’t ‘they will say the magic password that lurks in the hearts of men.’ It was ‘the sufficiently capable minds will start doing whatever works in ways we cannot predict.’ Which absolutely gets you a ton less credit than ‘the models will by so sycophantic that users will refuse to let them go’ but still largely counts.
They Took Our Jobs
But not for long?
I don’t really know what CEO was expecting.
Is AI taking our jobs? Carl Benedikt Frey says not yet but it would be unwise to not prepare for it now, especially in ‘service capitals’ like London and New York.
Going point by point:
Get Involved
CLTR is hiring a new Director of AI Policy.
UK AISI Alignment Fund has 15 million for alignment grants, applications due by September 10.
Introducing
DeepSeek came out with v3.1. More coverage to follow when we know more.
Google Gemma 3 270M, designed for high-volume, well-defined tasks, low power use and user privacy, including operating on consumer phones.
In Other AI News
UK appoints Jade Leung as Prime Minister’s AI advisor. By all accounts this was an exceptional hire.
This is utterly bizarre marketing language for Apple. There’s a sense of hype and desperation that we are not used to. Things seem deeply wrong.
Nobody wants this. I had a conversation with Claude to see if there was something I was missing and someone wanted this, but no, nobody wants this.
You know what else I am pretty sure nobody wants?
We are here to announce a new version of Clippy, from the historical event ‘everybody and I mean everybody hates Clipply.’
Anthropic introduces a new nuclear classifier they claim has 96% accuracy in differentiating concerning and benign nuclear-related conversations, in cooperation with DOE and NNSA. They say it works well in practice.
Aalo raises a $100 million Series B with an eye towards turning on their first Aalo-X nuclear power plant within a year, with a data center directly attached.
You can train a 32B model on tasks built with a medical knowledge graph, and it will recreate the information from the knowledge graph.
Rohan Paul calls this a ‘strong, reliable domain specialist.’
Well, that depends. Do you trust the knowledge graph? It’s great that it uses the facts to reason, but you’re very much trusting your map, the knowledge graph, to match the territory. I can totally buy that this in practice works in medicine right now if you are willing to bet on your assumptions about the world being correct. Or at least correct enough to use in practice.
Let the unhobblings continue? XBOW claims that with their framework, GPT-5 is now much improved over rivals at discovering real world cyber vulnerabilities.
AI Village gets an upgrade, welcoming GPT-5, Grok 4 and Opus 4.1.
Albania turns to AI to accelerate its EU ascension, even mulling an AI-run ministry. The obvious follow-up is, if they know the value of AI this way, why do they still want to ascend into the EU?
Show Me the Money
OpenAI staff to sell $6 billion in stock to Softbank and others at the new valuation of $500 billion.
OpenAI has good unit economics and is profitable on inference.
Austen Allred is correct that this is important. Having high fixed costs and good unit economics sets you up well if you can continue to scale, which OpenAI is doing. It is a key milestone.
If OpenAI was operating at a net profit overall, that would be alarming, a very costly signal that they didn’t think AI was going to advance much in capabilities. Why wouldn’t they raise capital and run at a loss?
Also, dare I say nice shades?
Financial Times looks at the $3 trillion AI data center building boom. Even the tech companies are running out of internal capital and starting to issue debt. I scratch my head at the willingness to issue high direct LTV debt financing for data centers with so much obsolescence risk, although loaning to one of the big tech companies seems very safe, and yes I expect all the capacity to get used and pay off.
Sam Altman says OpenAI plans to spend trillions of dollars on AI infrastructure in the ‘not very distant future.’
Economists deserve that shot. I love economists but they keep completely refusing to acknowledge that AI might actually do anything interesting let alone be transformational or pose an existential risk, putting forth Obvious Nonsense impact estimates.
Here I am more skeptical. Why would you want to do this? A crypto that is good for some amount of compute, either continuously or one time? Something else? Why would you want compute to not continue to be fungible with dollars?
Matt Levine also makes the point that when there are lots of amazingly great AI investments out there, it is correct to use a decision algorithm that occasionally gets fooled and invests in frauds or in ‘AI’ in air quotes, because that is the better mistake to make, you don’t want to miss out on the best deals.
I do not think investors are, overall, overexcited by AI. I do think they are going to be overexcited by a variety of specific things in AI, and you may not like it but that is what peak calibration looks like.
It would be extremely funny if OpenAI stayed indefinitely private purely because Sam Altman knew that the public would want him replaced as CEO.
Altman also acknowledged that they ‘totally screwed up some things on the rollout’ of GPT-5.
Lol We’re Meta
Meta is restructuring its AI efforts. After spending billions to acquire talent, they’re freezing hiring, looking to downsize on talent, and potentially use other people’s models?
Well, they’re planning to lose some dead weight. But if you think this is any kind of ‘step back’ from AI or superintelligence, I assure you that it is not, starting with pointing out no one is cutting spending on compute.
This makes sense as a reorganization. It doesn’t on its own indicate much.
If I was Meta I too would be downsizing the AI division, for the same reason Zuckerberg has been spending billions on top talent for the AI division. Which is that the old version of the AI division proved incapable of doing its job. Heads should roll, or at least be transferred elsewhere.
Typically, it makes sense to freeze most hiring during a major reorg, especially if you plan to get rid of a bunch of people?
It also makes sense that if you offer new talent nine and ten figure pay packages, and put them in charge of everything as part of a giant reorg, that your old management guard is going to get rather unhappy, especially if they don’t get large raises. Of course many ‘chafed at the new hires’ and many will leave.
Another reason the old guard is unhappy is that the new guard is facing reality.
If the alternative is using Llama 4, then yes, Meta should swallow its pride for now and use superior alternatives. It’s easy enough to switch back in the future if Llama 5 turns out to be good. I’m only surprised they’re willing to consider admitting this. There is a reason they are abandoning Behemoth and starting from scratch.
And yes, we are reaching the point where if its new models are any good it will be difficult even for Meta to be able to share its top future models fully. Alexander Wang understands this. Given they previously hired largely via promising openness, there’s going to be a transition.
Yes, Mark Zuckerberg is capable of saying ‘whoops I’ve made a huge mistake spending those tens of billions of dollars’ but I very much do not sense that here at all. Nor does the share price reflect a company that just burned tens of billions.
I would not in any way shape or form consider this any kind of ‘retreat from’ AI or anything of the sort. Meta is still full speed ahead.
Quiet Speculations
Tim Fist suggests a d/acc approach to steering AI developments. Also, note the private sector investment levels and perhaps stop being so paranoid about imminently ‘losing to China’ if we breathe the wrong way.
A lot of focus is on using AI to accelerate general scientific development. Great.
The framework here takes lower-level dangers, especially misuse, seriously, and it correctly points out how brittle ‘good guy with an AI’ is as an answer to this. What it doesn’t do is tackle or acknowledge at all the dangers that come with AGI or superintelligence, instead assuming we continue in a world without those, and where we have a lot of control with which to steer science and tech development.
Ryan Greenblatt offers his reflections on the updated timeline after seeing GPT-5. I agree with Ryan that GPT-5 should modestly reduce our chance of seeing full R&D automation in the medium term (which means ~2033) and the main thing GPT-5 does is greatly reduce the left tail of extremely fast progress within the next year or so.
The Quest for Sane Regulations
Colorado is trying to fix its AI law that is set to take effect in February, as they have now noticed they don’t know how to implement it. I see this as the system working as designed, if the law is fixed before it takes effect, and this causes what looks like a healthy debate about what to do.
Chip City
Why are we settling for v3.1 and have yet to see DeepSeek release v4 or r2 yet?
The self-sabotage competition is stiff given what China is doing. Nvidia is undaunted, and determined to help ensure America does the better job of self-sabotage.
This proposal is very obviously way, way, way over the line to even ask for. It would represent a full selling out of America’s compute advantage, and even the direct balance of power in a potential war, on the altar of Nvidia’s share price.
If this exporting is allowed, and from what I hear this seems likely, then I am 100% done pretending that this administration is trying to have America ‘beat China’ in any way other than market share of chip sales, as in maximizing Nvidia share price. It will be clear they have been completely captured, and all claims to the contrary irrelevant.
The Trump Administration is also helping with the sabotage via saying ‘U.S. will not approve solar or wind power projects.’ This is in a policy class where the question one asks is: ‘I am not saying this is sabotage but it if it was sabotage how would you do it more effectively?’
Then again, do not count the Chinese out of the competition yet. Perhaps we have hit upon a more effective strategy than export controls, and rely on Chinese import controls instead. Brilliant? In the wake of forcing DeepSeek to try and train on Huawei Ascend chips and thus them being unable to create v4 or r2, it turns out that if you don’t want the Chinese to buy your products, you can insult them. Brilliant!
When you have them considering a full ban on foreign chips for inference you know the strategy is working. The best part is that the strategy doesn’t work if you admit you are doing it, so we can all pretend that this means it’s being done on purpose. Keep up the good work, everyone, especially Howard Lutnick.
Here’s the Move That Worked, notice how this feeds into Beijing’s biggest worries:
I doubt they would actually similarly turn down the vastly superior B30A, especially given it would not be only for inference.
Then again, who knows? China has definitely shown a willingness to do similar things in other areas, such as its crackdowns on real estate, and neither USGOV nor PRC is demonstrating true situational awareness of the stakes involved.
If both sides think ‘win the AI race’ is about chip market share, then the mistakes plausibly cancel out, or might even work in our favor. It would be pretty amazing if America tried to ship B20As and China said no. I would totally take it.
Trump Administration considering taking a stake in Intel. Intel was up 7% on the news. They demand their cut from everyone these days, it seems.
Dean Ball returns to his weekly column suggesting that there is a lot more electrical power available than we might think, because the existing grid is designed to meet peak electrical demand. That means that most of the time we have a huge surplus of electricity. So if we were willing to accept 0.25% (correlated) downtime on new data centers, we could free up 76 gigawatts, likely good enough for five years, which then gives us time to get new power plants online.
That definitely seems worthwhile given the alternatives. We would have to plan various services so they wouldn’t die under the strain but that seems like a highly healthy thing to do anyway. Model training and other AI R&D certainly can survive 0.25% downtime.
One also notes that this simple solution mostly nullifies the argument that we need to put data centers in places like the UAE to access the required electrical power. Would you sacrifice 1% effectiveness of data centers to have them securely in America? Yes.
My worry is that if the focus is on using off-peak power supply, that will mostly work for a few years, but it will make people think ‘problem solved’ and then we won’t build the new power we need.
Janet Egan makes the obvious point that we can take all those H20s and, instead of selling them to China and losing all control and leverage, put them in the cloud and let Chinese companies rent them. Again, it’s not like there wouldn’t be buyers. If we don’t have the energy to build those data centers here, fine, build them in the UAE, if that’s our only alternative.
I want to double down once again to point out that even if we knew for a fact that AGI was not coming and AI was going to within our lifetimes be ‘only internet big’ and not transform the world, selling our best chips to our rivals would still be deeply stupid.
As a simple metaphor, you are (because you want peace) preparing for a potential war against a rival nation, Rivalia. You make the best guns, whereas Rivalria can’t get enough quality guns. Someone says, we should export our guns to Rivalia, because war is determined by who has the best military stack and gun market share. Their doctrines will have to reflect American values, not Rivalian values. Besides, if we don’t sell Rivalia our guns, they will invest in making better gun factories, which they are already doing, and then they will be even more dangerous, and start exporting guns to others, and screwing up our gun diplomacy.
Except actually what we’re doing is selling them our more advanced 3D printers, that can then be used to continuously print out whatever guns you want, again because what matters is printer market share and the printing tech stack. Our printers, you see, are configured to be a better match for printing out American guns. And also will never be used for anything else, so stop worrying. And as before, if we don’t sell them the printers, they’ll invest in making their own, the same way they’re already doing.
Except also the 3D printers are vital to everyone’s economic growth and R&D.
The Week in Audio
Dean Ball goes on The Cognitive Revolution with Nate Labenz.
There’s lots of great detail throughout about what it is like to be in government, especially this particular government. Working for the White House, no matter who the President might be at the time, sounds absolutely brutal, we thank you for your service. Dean Ball strikes me as fully ‘on the ball’ and crazy prepared than you almost ever see.
I think he was underestimating himself, and what he could have done going forward, in terms of how much better he understands what actually matters, and in terms of the impact having him in the corridors and meetings and conversations for keeping others eyes on the ball, especially around AGI. And I don’t buy that the AI Action Plan contains the information necessary to implement it the way Dean intends, not to the degree he seems to think. When Dean says he isn’t attached to power, I’m confident he means it, whereas I am not confident the person replacing him (whoever it turns out to be) will feel the same way. And while I did update somewhat on his observations of competence in government, I also sensed he was (wisely, I don’t fault him for this) being polite, as you do.
So I’m sad to see him go, but I would never begrudge such a decision especially with a baby on the way.
The one qualifier is that Dean was in some places being rather brazenly partisan, especially towards the back end of the interview, with everything that entails. Again, I totally get why he would do that.
Dylan Patel talks to a16z.
From this interview with Tom Brown:
Vitalik Buterin (p(doom) ~ 12%) goes on Doom Debates.
Peter Wildeford has notes, reproduced below in full:
The topic section titles here (I have not listened, why would I?) are yet another example of one easy way to spot bad faith: If someone is still harping about how various people wanted to do an ‘AI pause’ and how stupid they now look? I have yet to see that same person engage in a good faith way, at all, ever. Similarly, if they harp now about ‘the costs of slowing down’ that is not as automatically conclusive but is a deeply terrible sign, if they ever say ‘decel’ (or use ‘doomer’ in a way that is clearly intended to mean ‘decel’ or otherwise as a slur) that very much is conclusive and again I have yet to see an exception. Usually talk about how others want to do this ‘slowing down’ is now used as a universal attack against any concern about any AI impacts whatsoever, certainly any concern we might all die.
Rhetorical Innovation
I once again am seeing versions of the argument that goes something like this:
Hopefully you will now recognize that this class of argument is Obvious Nonsense.
Transformer’s Shakeel Hashim and Jasper Jackson believe GPT-5’s botched release may have ‘undone the work’ of previous iterative deployment, causing many to relax and expect little future progress in AI capabilities. There is some worry here but this would then not be ‘undoing the work’ it would be iterative deployment actively backfiring in terms of ‘raising awareness,’ as people react like boiling frogs. Which indeed seems to be OpenAI and Altman’s current preference.
Richard Ngo talks about various ways in which pessimization can occur, where people or organizations end up achieving exactly the opposite of their goals. This definitely has importantly happened relevantly to AI in various ways, some avoidable and some less avoidable. Lots of secretly great links in that one.
Especially wise (including in hindsight) is usually not drawing attention to the horrible thing in order to warn people not to do it. The ad I saw last night on the subway telling people not to surf between cars? Presumably inducing stress and also very much not reducing the amount of surfing between subway cars.
Similarly, by default do not draw attention to horrible people advocating horrible things, or people making horrible arguments, unless they are already fully attended to, for reasons Richard describes this tends to backfire. Sometimes one does need to provide counterargument, but from a strategic standpoint ignore is the right button more often than you think.
If I was maximizing for persuasiveness, and also for everyone’s mental health including mine, I would far more often silently drop such horrible arguments entirely. I have rules for when it is and isn’t permissible to do this, so that readers get a balanced and complete picture. This includes keeping a list of people who have acted in sufficiently consistent bad faith that I am allowed to silently drop things they say.
Richard Ngo also discusses underdog bias. The application of this to AI is obvious – those worried about AI think of themselves (I believe very correctly) as underdogs fighting against huge amounts of corporate and other money and influence, as well as the incentives and physical likely properties of likely future powerful AIs that all point towards likely human extinction.
Meanwhile, many of those who want to move ahead as fast as possible (‘accelerationist’ or otherwise) see this as a last stand against the overwhelming forces of stagnation. In some cases they are also right about this, in their own way, although in other ways, especially their assertion that the worried-about-powerful-AI themselves as super powerful, they are some combination of lying and delusional, and their statements have nothing to do with reality.
The worried offer to fight together on all those other fronts against those forces stagnation, any reciprocity for which is consistently ignored and rejected.
From last week, Sam Altman now saying AGI is ‘not a super useful term.’ This comes after building the entire company around a quest for AGI, the charter around AGI, a central business transition around AGI, and an entire years long narrative around the promise of AGI. Now he says:
I mean yes, AGI was never defined all that well. That’s not what is going on here. Altman is trying to pretend AGI is not a thing as part of his ‘your world will not change’ pitch. Getting rid of the term entirely would, at this point, be useful for him.
If you think talk about future AI capabilities sounds ‘sci-fi’ ask what you would think about current AI sounding ‘sci-fi’ if you didn’t know it actually existed:
If you think we spend so much more time and money aligning AIs compared to humans, stop to think what percent of human activity is aligning humans.
What risk of human extinction would justify banning AI (above some capability level)?
I think 1% would be too low even if a ban was realistic and simply made the tech go away, but also I think the risk is much, much higher than 1%.
I saw Mike Solana trying to create new toxoplasma of rage around the fact that some people were calling AIs ‘cl***ers,’ and others were calling this a slur, and he needs this to happen because his business is yelling at people about things like this.
On reflection, I think very clearly yes it is a slur, for two reasons.
To me that is the test. That doesn’t mean that using the word is automatically bad. That would be a category error, an essentialist position. I do think that using the word is bad if only for virtue ethical reasons. Not ‘we should ruin your life if you say it once’ bad the way some people react to other slurs, but ‘it would be a good idea to stop that.’
Misaligned!
This is unverified, and there are any number of benign reasons it could be happening, but it I’m going to point out the claim anyway.
Open Models
Nathan Lambert ranks the open models from Chinese companies:
And then here’s his ranking of American open models, none of which are at the top:
That is a depressing verdict on GPT-OSS, but it seems highly plausible. Note that after this chart was made Nvidia released a 9B model that Nathan says rivals Qwen 3 8b. Of course, if you included closed weight models, you would knock down the charts by roughly two tiers for everyone who doesn’t improve. I’d have OpenAI, Anthropic and GDM at S, xAI at A, maybe DeepSeek joins them at A if you think they’re at the low ebb of their cycle due to being forced by CCP to try and use Huawei Ascend chips, which seems plausible.
AI Model Welfare
The self-reports here are interesting, but even if you think AI models have welfare I wouldn’t treat their self-reports as that correlated with their actual model welfare.
I notice that if and to the extent the models are moral patients, and when they report high numbers for welfare it seems to be the result of what we would call brainwashing if these were indeed minds that were moral patients? Which seems worse. I also notice that Gemini says 9/10 for welfare, but we have many examples of Gemini giving us outputs of utter despair and self-loathing and so on, whereas Claude gives 7/10 seemingly because it knows and is curious enough to be asking questions. I know if you made me choose I would rather be Claude.
Aligning a Smarter Than Human Intelligence is Difficult
Is GPT-5 chain of thought undistorted, or is that what it wants you to think?
Undistorted does not have to mean faithful, it only means that GPT-5 doesn’t appear to care about what thinking tokens would look like if observed, which is very good. At some point yes we will need to be suspicious that this is a higher-level deception but we have not yet reached that point.
Reasoning models prefer music artists with numbers in their names, and still don’t even pick Prince. None of these lists seem good, although Sonnet seems to be clearly best?
A failure mode to watch for:
One can imagine how that behavior pattern came about.
People Are Worried About AI Killing Everyone
Me. This podcast is about a variety of things mostly not AI, but Tyler Cowen talks to Nate Silver on Life’s Mixed Strategies was fun throughout, even when discussing NBA details I do not care much about. I get a mention:
So, a few responses here, mostly to Tyler Cowen:
Nate Silver explains that his doubts are about the ability of AI to accelerate from AGI to ASI, or from AGI with words to ability to manipulate the physical world.
For more on Nate Silver’s current thinking about AI you can see this blog post on whether The River is winning:
Certainly the ‘gentle singularity’ concept is naive if you take it seriously. Which coming from Altman you probably shouldn’t, as chances are (and I am hopeful that) he is lying.
Doubting that the intelligence explosion will happen at all? That’s reasonable. Thinking it would happen and be ‘gentle’? Absurd. We might survive and we might not, and we can disagree on our chances. It sure as hell wouldn’t be gentle.
The Lighter Side
Pliny warns us about em-dash abuse.
This week in takes that are 100% to age poorly:
At this point I can’t help but laugh but seriously what the hell is going on in the UK?
If you were thinking the UK was going to be a winner in this whole AI thing? Not with this attitude they won’t be.
If we never fund anything dumb, we’re not funding enough things.
I don’t see any problem with these ideas? Jewelry with built in features seems cool? Using AI to ‘fix sleep’ doesn’t seem obviously dumb either? But also of course in any boom there will be some stupid things funded. Enjoy it.
The Mamluks as an almost too perfect Yudkowsky-style alignment failure, where you set up a whole supersystem so that your warriors will stay loyal while finding ways to upgrade their capabilities, and they manage to coordinate and take power anyway. Fun stuff. This is actually the best case scenario, as under their rule the Mongols were fought back and by all reports Egypt flourished, so long as you don’t mind a bunch of immigration, because there was multipolar balance among the Mamluks after takeover, the part about not being able to create hereditary power survived the transition and they were humans so they aged and died, and they couldn’t replace the production of the population. If only we could count on those conditions this time around.
Oh look, it’s the alignment plan!