It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year.
Another key leak this week was the draft bill Anthropic had been shopping around Congress, similar to but weaker than the FRONTIER Act. Anthropic’s initial attempts to explain itself only made the situation seem worse, as they seemed to not understand the problem with what they had done. Dario’s later response was better, but the situation remains troubling. There are enough different Anthropic developments, that I plan to have a post soon covering them all. It is a mix of news running from excellent to terrible.
Will interpretability save us? There is a new claim that neural networks are cleaner than we thought, meaning that individual neurons mostly have a single interpretable meaning, and it is tractable to go through and label them one by one. Initial reactions got out of hand, treating this as a far bigger development than it was. This is still excellent news that makes our estimate of difficulty substantially lower going forward. The work can begin.
I continue to owe a write-up of the Gemini 4 Argon Safety Card. That is in progress and hopefully I will finish it soon. Life has gotten in the way. I may or may not have a full capabilities report on Gemini 4 Argon once I tally up the reactions. Muse Wonder definitely will not have such a post, as it has proven to have been oversold.
58% of Anthropic staff expect Claude to make a Nobel prize level scientific discovery by 2027. I agree with Roon that this should have been over 95%. The event itself might not be over 95% to happen, but it has to be a clear favorite to happen, so it is crazy for so many employees to not agree with that.
Tell you what you actually want to know straight up.
PoliMath: Me: How long should I bake chicken wings
Google top result: “When I first proposed that we start a backyard coop, my husband was deeply reticent (5 pages later) Anyway, here is our recipe for honey mustard chicken wings”
Robin Hanson: Whenever you put AIs in charge of something important, you have to wonder how the firms which make them will respond to various pressures from powerful parties to slant their AIs in particular directions.
The point of a jury is to provide a check on the system and avoid abuses or overly strict adherence to the letter of the law, and to protect the innocent and our rights, not to be accurate. Judges are a much superior accuracy tech. Thus, an AI jury (or one AI judge) does not solve your problem. What you likely want is a human jury that has the ability to query AIs. They can override that, but have to do it on purpose.
If you read AI writing, you can then talk to the author in some sense, or read their further thoughts, but not in the sense that Tyler Cowen would like.
DeepSeek-V4.1-Flash-1017 exists. Their benchmark chart contains Opus 5 and 5.6 Sol. It’s probably a solid upgrade over the old V4.1-Flash anyway, but wow.
How did it come to exist? One guess.
Lisan al Gaib: very curious that DeepSeek now has the same terse reasoning style as Astra. must be a coincidence.
Dean W. Ball: Distillation has always been part of the explanation for why China is so close behind the U.S. in model benchmark performance. But one of my updates after joining openai is that distillation probably plays a bigger role in the explanation than I realized.
I haven’t really shared this publicly because people will next-token complete the obvious cope “buhhhh you are just saying that because derrrr” replies but it really is true, regardless of whether you think it’s good or not.
I would have guessed distillation explained ~30% of Chinese close-behindness before seeing what one can see within openai, and after that, more like 50%.
Muse Spark Open exists, as in a version of Muse Sparkis now an open model. They call it ‘the most capable US open-weight model to date.’ That seems likely to be true. It scores a 39 on Artificial Analysis. The open weight frontier for that index is 46 for MiMo-V2.6-Pro and Qwen-3.8, and MiMo is a lot cheaper to run than Muse Spark. Nathan Lambert calls this his ‘super bowl,’ says this closes the gap substantially, and points to the Interconnects Adoption Dashboard where Spark is #8. This is not a fair competition, since the Chinese can do Massive Fraudulent Distillation, whereas Meta does many things but it won’t do that.
There are some advantages to having a semi-competitive American open model. I wouldn’t say that I trust Meta, but you can use their models in places where you would not be comfortable running Chinese models, even locally.
Lack of upgrades: There was no major Chinese open weights release between September 21 and October 16. No DeepSeek, GLM, Kimi, MiMo or Qwen.
Nathan Lambert (I asked him why no releases): Compute restrictions, I do think there’s more internal lab-govt discussions on safety in China, less organization development. No good answers, but I’ve been monitoring & pondering
We then got a new DeepSeek, but no version number increment, on October 17.
One theory is this is a random lull. Another is that Anthropic and OpenAI improved their distillation defenses, and the Chinese are stuck.
muse wonder is the best model in the world for personal agents and professional agents.
we evaluated our model against the best models in the world across agentic and coding tasks, and we think you all will like it.
Some quick responses were very positive.
Eleanor Berger: Used the not full version for only a little bit and it’s definitely very good. Like previous Muse models, it’s better in written text and interaction than all other models. And it’s very fast and smart and agentic. Time to start taking Meta seriously as a frontier AI lab.
Theo (this was before Argon was available): My current model rankings
S – Opus 5.5, Muse Wonder
A – GPT-6.1 Sol
— Gap —
Everything else.
Kelsey Piper: I think it’s more narrowly a coding model than the latest Claude and ChatGPT offerings. It’s markedly worse (both on benchmarks and in use) at research and writing.
Same as it ever was. Meta is making progress, but is still solidly behind the frontier.
Meta also reinforced its reputation for overselling and being untrustworthy. Once again, they have misrepresented a new release. Claims of things like ‘best model in the world’ for personal agents and professional agents’ are clearly false.
On Your Marks
Kelsey Piper on some current AI writing capability failures:
Kelsey Piper: Even frontier models have poor theory of mind, they suck at having characters know something and not say it without the narration directly telling to to you anyway. This is crippling for espionage/detective stories in particular
They’re very mechanical when trying to convert one story to a different format, or else have bad judgment about which details need including. Salience/importance is hard for them.
Their characters all talk the same or else like exaggerated caricatures of themselves. They’re really bad at the kind of subtle humor that makes the infodump parts of eg Tom Clancy keep from stalling out
Win rates for Ascension 0 are now very high, yes ‘top human’ level is close to 100%:
Ascension 20 (max difficulty) is much harder. You do not win ‘by accident’ or by taking generically good cards. This is a higher win rate than I expected to see so soon and Sol and Astra are doing much better than the average player, even confined to those who unlocked max difficulty and choose to play it.
Slay the Spire 2 also sees the models struggle at max difficulty (A10), but yes it is easier right now and you can sometimes get lucky.
Is this a coincidence or is it deliberate training? The working theory is a mix of both.
Thomas Friedel: do you think OpenAI had a RL env to train Astra specifically for STS ? Performance is unbelievably high. Higher than my dedicated STS2 model.
Benjamin Ou: Benchmark creator here! Probably yeah, GPT generations have gotten ridiculously strong at this while not necessarily being as strong on easier games like EBR-bench, and other models are pretty far behind. We know the labs like the game, so all signs point to RL.
Not to downplay how strong they’re getting in general at strategy card games other than StS though, there’s def some generalization going on as well.
I continue to see many reports like this, whereas months ago people said the opposite:
Utah teapot: kinda weird how a week worth of gpt 500 seems to get about as much done as 5 hours of claude 200.
Get My Agent On The Line
UPS blocks AI agents. So do for example Delta, JP Morgan Chase, Charles Schwab and Amazon. The group doing this by default is going to keep expanding. The agents will still be useful, but less so.
The biggest other practical barrier for agents like Dots, and also many new Claude projects, is overrefusals and misclassifications that require permissions. This is a hard problem, since the other error is worse. As an example, I gave my Dot a ‘do not delete anything that does not have source control’ rule before trying anything else, and it took that to some rather extreme places. Claude auto mode asking for permission so often is actively less safe than if it asked less, because many users will quickly stop even reading the requests.
Patrick McKenzie: It is deeply weird to, in the middle of extremely productive work, have a long argument with a coordination process attempting to argue with a subagent.
The exact contents of the hour-long argument I will not recount, but at about thirty minutes in I was wondering how anything capable of writing software code could be so obtuse. About an hour in I thought “Oh I am not first ~CEO to have this thought am I.”
Deepfaketown and Botpocalypse Soon
In practice deepfakes continue to not be a substantial practical problem. Politicians still care if you enable them. It is one of the easy bad things to understand.
Chris Sommerfeldt: [NYC] Council Speaker Julie Menin tells SpaceXAI’s Gabriela Castillo Madrid she created a sexualized image of herself using Grok. She asks how that’s possible if Grok has sufficient controls.
“I’m happy to have our safety team look into that specific instance,” Castillo Madrid says.
After the hearing, Menin says she’ll “definitely” follow up with SpaceXAI about this because she wasn’t satisfied with the testimony. She also says Castillo Madrid was “lying” when she claimed she didn’t know about the firing of a SpaceXAI employee who raised security concerns.
What happens if AIs can launch frivolous lawsuits? Defense is far from free.
Ethan Mollick: A consequential choice will be made soon by government and companies as they are increasingly flooded with agentic AI inquiries that overcome the frictions that gated access (see the graphs – it has already started!)
Two options: do you make services worse or better as a result?
Status quo isn’t an option. You can either add additional frictions (adding fees, make services worse, ban agents) or remove them (provide more services with AI help, redesign work to make access easier). Paper [here].
Flo Crivello: Seeing this in my circles — several founder friends have told me they’ve started getting hit with a huge amount of frivolous lawsuits
Dean W. Ball: It’ll be interesting in particular if people feel they can self-represent when launching frivolous lawsuits. Lawyers can be punished or suffer reputational damage for truly egregious frivolity, but a non-lawyer cares less about these things.
You do both. The good news is that there are ways to ‘make it worse’ that do not do that much damage to legitimate users.
Yes, some people are lying scum who will present AI-written stories as real in order to drum up outrage. This is a demand-driven problem. The AI makes faking the story easier, but it also makes the fake stories easier to catch. Blame the human, not the AI.
Kelsey Piper: Half a million views, tons of right wing posters like this one going “when we talk about cultural compatibility it’s not some hypothetical issue”, and no one asked “is that letter entirely AI generated” (yes) and “is it in fact almost the exact letter you get if you say ‘generate a fake HOA letter telling me to stop smoking brisket’ (yes), “would a first class letter dated Saturday October 17 arrive in time to complain about it bright and early Monday morning” (no), “is the letter consistent with Arizona laws around notice periods” (no), “how many HOAs in Arizona have a nuisance provision 8.4” (5, according to ChatGPT), and “do any of them employ a Marjorie” (no).
Fun With Media Generation
I continue not to understand how restaurants end up using AI images of food when they have real food you can photograph. Do people actually prefer it this way? I suppose many of them do.
Spy Versus Spy
Update on the OpenAI spy: Yes he used his wife’s ChatGPT to plan planting the bug. There were at least two recording devices. He was in communication with a member of the Saudi royal family and there are additional clear signs that his client was the Saudi government. There’s also this:
Kevin Khadavi: My investigation into the alleged OpenAI spy’s Stanford ties got 126k views, but the response seemed unusually negative. I decided to look into it: the overwhelming majority of accounts attacking my reporting are Saudi.
I like to think that Anthropic employees get similar training, except they also are told not to talk about it. If not, get on that, guys. Same for the other labs too.
unusual_whales: BREAKING: OpenAI is warning its employees to watch out for spies, especially romantic ones, per WSJ
The company has cautioned staff that someone trying to seduce them may likely be a “honeypot.”
Even strangers approaching them in everyday places could be gathering info.
Georgia Wells, Erin Woo and Meghan Bobrowsky (The Wall Street Journal): One story OpenAI tells employees to illustrate the risk: To seduce a U.S. diplomat in Russia, a foreign espionage operation first sent a blond woman to his hotel room with Champagne. When he declined, the spies sent a redheaded woman, and then next a young man. The implication for OpenAI employees was this could happen to them, according to the people.
I like to think that if I was a diplomat, by the time I see the second unknown hottie show up at my hotel room with Champagne I would be pretty confident it was a trap. There are many things that are much more effective on their first try. Mix it up a little.
Kelsey Piper: I worry this makes things really hard for extremely attractive people who just want to seduce an OpenAI or Anthropic employee for their money.
Kelsey Piper: imagine being engaged in the innocent and time-honored trad pastime of marrying the richest guy you can find and being accused of international espionage just because you are really really hot.
alice: i personally want to seduce them for their extremely accurate meows
What changed is that we noticed the problem. The problem has existed for a while.
Zack Korman: The OpenAI employee who allegedly stole over a hundred confidential documents without OpenAI noticing for months sent the documents to his number *in mainland China*
This is all a big problem in its own right. If you cannot protect your secrets from spies, including the Chinese, then there is no point in trying to ‘win the race.’
This is also indicative of other problems. If you cannot solve your employee misalignment problems, how are you going to solve AI misalignment problems?
A Young Lady’s Illustrated Primer
The first best way to learn with Claude is very different from ‘create practice sheets and teacher updates’ but it is a vast upgrade on similar actions without Claude.
cold: Kids these days are getting tiger parented by Claude
I’m feeling out a Take that if a tutor is not using AI in their work at this point, they are veering towards negligence, because tutor+AI now dominates tutor alone at ~all levels of competence
Vladislav Ekimtcov: claude obsoletes school tbh. you can do an entire k-12 in a single github monorepo with fun little widgets that teach how to write every letter and add up numbers
Jack: Five years of AI has changed the world dramatically more than five years of electricity or five years of the combustion engine did. It’s not even close.
AI so far has simultaneously changed the world more than any other technology (at least if you exclude military techs that won active wars) in its first five years. AI so far has also changed the world dramatically less than I or many others would have expected in that time, if we had been told the path of AI capabilities. Diffusion takes time and a lot of things have been bottlenecked.
As AI use continues to grow exponentially, expect bigger impacts, with the impact growing faster all the time, even if capabilities mostly stop improving from here.
Anthropic is hiring for a smaller share of junior positions and AI agent hours outnumber employee hours six to one. Human attention is a bottleneck for AI outputs and human productivity is way up. That’s the central finding of a new Anthropic report on what it is like to work there.
Anthropic: At the same time, the overall employment share of technical departments within Anthropic has fallen, from 61% to 56% in the last year. As the company expands, technical hiring has continued growing in absolute terms, but intensive hiring in sales, finance, and recruiting has outpaced it.
… Even clerical work tightly connected to in-person interaction has grown slowly. An executive assistant, who built a personal assistant agent for scheduling and email triage, explained: “I think we’ve already moved past the point of Claude augmenting away the really entry-level admin type role,” and instead focused their work on higher-order “chief of staff” tasks.
These changes have already involved reallocation: employees working at the company for at least a year were 50% more likely to move across departments each month in 2026 than they were in 2025.
They have a chart for ‘reading Claude as a full time job.’
Some think that soon either the jobs be gone or the jobs be fake.
EigenGender: There is simply no future where humans continue to do economically useful work for much longer. Your choice is at best between various flavors of “ASI let’s humans have fun and play make believe.”
(to be clear this is a good thing!)
Opinions are very strong on whether or not this would, if it happens, be a good thing. In some ways it is very good. In others it is very bad.
Anatoly Karlin: I strongly believe that everyone countersignaling against post-scarcity utopia on the basis of the “dignity” and “sanctity” of work should have do a lengthy stint as a Dhaka garments factory or African subsistence farmer. Talk is cheap from the air conditioned ivory tower.
Jack: I very strongly want a world where humans continue to pursue meaningful expertise and accomplishment, but I think Karlin is right that a lot of dignity-of-work takes come from people who have the uncommon luxury of dignified work.
Some view not having real work as being kind of AI pets (you are the pet, pejorative).
Linus Mixson: I suspect it’s as non-obvious to most people as it is to me that “ASI pets” is a more desirable future than is extinction, and I think it’s worth straightforwardly confronting the fact that Anthropic, both as a collective and at the level of its individual constituents, seems to think it rather decisively is.
Aleph: This has always seemed obviously wrong to me. If you ask the average person if they would prefer they and their family be murdered to living forever in a world of AI abundance they’re going to laugh at you. Absurd to believe otherwise.
Most talk of ‘pets for the AI’ is nonsense, but the answer to ‘does humans continuing to exist in abundance (or even adequacy) without needing to do productive work constitute a net positive future’ seems to obviously be yes. It might not be the ideal path, but the hard part is making that future have an equilibrium of that type, not coming up with net positive experiences.
Get Involved
You can go work for a Congressman on AI. Seems impactful.
Experience in AI, policy research, or national security is encouraged.
Not AI, but if you know anyone who wants to hire Terese, she’s awesome.
Terese: I’m looking for work as a wet lab biologist in the San Francisco Bay Area. My PhD was in genetics and neuroscience but I’m open to exploring new fields. I’ve only ever been in academia, so if you do wet lab work in biotech, I’d love to hear what it’s like! DMs open.
Jakeup: I dragged my wife to SF from a post doc at a Nobel laureate’s lab at Columbia with the promise of cool Bay Area biotech. Cool Bay Area biotech, this is your chance to snap her up.
Citrini Research goes to Anthropic’s offices to try and tell Claude to do a medical breakthrough, make no mistakes. Early results are promising and they conclude the frontier has moved, but you still have to test the ideas the old fashioned way.
Microsoft publishes a detailed report on how they did a new 3T-sizepre-training. Their presentation includes chart crimes. This seems like a ‘those who know shouldn’t tell and those who tell don’t know’ style of report. Claude’s analysis is that they are still not so close to the frontier, but that this is plausibly something they can usefully license or use for recruitment, and that it has minimal uplift value.
SpaceX shares are trading around $146, so if all the shares are transferred Elon Musk could be granting around $10 billion. That is real money.
Giving the shares to the public sounds cool. It does not make economic sense, and the transaction costs will be high, and it does nothing to make us not die despite Musk thinking we might all die. It still might be the best option.
The reason you do it that way is symbolic, and to maybe give people the idea that they are ‘invested in’ SpaceX’s success. If that’s what gets him the motivation to do it, and how it can happen and avoid government taking a stake, then great. Second best help is still help.
Returning to Anthropic, my biggest concern is that the Benefit Fund, like many employees, will not focus on making AI go well and having everyone not die, and instead choose less impactful causes. The only action they rule out is ‘lobbying or political giving.’ Yes, offering funds to buffer against economic disruption is a noble cause, but a secondary one, and trying to fix that privately is an infinite money sink.
So how did people react to the announcement of one of the largest charitable commitments in human history (my guess is it is third but there are five all between $150 billion and $225 billion)?
Kevin Roose (out of order for dramatic effect): this post announced what could be the largest commitment of philanthropic capital ever made ($200 billion if Anthropic goes public at $1 trillion, which is probably conservative)
by my count, if Anthropic and OpenAI each go public at $1 trillion (again, probably conservative), that’s roughly $460 billion, or 5 Gates Foundations, earmarked for philanthropy-shaped things
time to start those 501c3s y’all
there’s a funny bit in the book about the Anthropic cofounders reacting to Dario asking them to pledge 80% of their equity to charity
Jack Clark considered bargaining down to 50%, but Tom Brown told him “this will either be a trillion-dollar company or a zero-dollar company”
the first reply I saw asked if Anthropic was going to use it for “gay race communism”
The replies have settled to a more reasonable place, but there is still a lot of skepticism out there about how this money gets spent. Some are more skeptical than others. It seems totally fine to reserve judgment until one sees what the money will be spent on. Someone saying ‘public good’ does not mean their spending will advance the public good.
It still seems like a telling reaction if someone pledges to give away $200 billion and your first instinct is that they are probably up to something.
Bubble, Bubble, Toil and Trouble
Current American GDP is around $32 trillion.
Kalshi: BREAKING: Anthropic says “AI could push US GDP to $44 trillion by 2030”
70% chance of IPO this year.
Elon Musk: If this is rephrased as “useful output of US”, then >50% growth by 2030 due to [AI] is extremely likely, unless there is a major war
The word ‘could’ is often used to do a lot of work. I would say that if US GDP gets to $44 trillion by 2030 and this is mostly real growth rather than inflation, then you should be freaking out rather quite a lot.
Quiet Speculations
Scott Alexander writes Against Against Handoff, as in handing off control to an aligned superintelligence or superintelligences, once such a thing is available. His central point is that all the known alternatives are worse, and your imagined ‘we will have the dumber humans in control indefinitely in the ways we need to be free to do what we want and in control, but not in a way that lets a faction of humans or AIs take over’ plan has not been properly thought out.
OpenAI CEO Sam Altman lists three proposals for navigating AGI. This is a very narrow range of possible options.
OpenAI CEO Sam Altman last week told employees about three proposals the industry has discussed for how to safely develop AI, a stark framing that comes just as the Trump administration sets up its first system for policing “superintelligence” developers like OpenAI.
One path involved developing an international organization, similar to the International Atomic Energy Agency, that would enforce safety standards, a scenario Altman appeared to support, said two people with knowledge of the comments.
Another option would involve merging all of the AI labs into a single body to develop new AI safely.
Altman described a third scenario—which he didn’t prefer—in which one AI developer would obtain enough economic power that it could deprive all other developers, including those that could be dangerous, of computing resources, ensuring that they couldn’t catch up, these people said.
“Obviously I think that one’s terrible,” Altman said, adding that he was including it for “completeness,” according to a person with knowledge of the comment.
… While the first two options Altman shared with staff would require an immense amount of coordination between companies or governments, Altman said OpenAI can’t afford to give up ground to rivals at the moment, these people said.
… But in the next year, OpenAI needs to decide which of these paths it wants to push for, Altman said.
… Altman referred to the third option as the “Eliezer plan.” … a “pivotal act”
For true completeness, option four is a free-for-all, and option five is don’t build it.
Quickly, There’s No Time
Here is a good track of AI speedups.
SemiAnalysis: NVIDIA engineers at the PyTorch conference just said that the speed of optimizing CUDA software with Frontier AI models has increased rapidly.
October 2025 to August 2026: 5x in 1 year with expert human optimization
September 2026 to October 2026: 2x in 1 month with AI assistance
October 2026: 1.5x with AI orchestration + MegaKernels
The Quest for Sane Regulations
OpenAI writes to Governor Gavin Newsom asking for strengthening of SB 53, including AI R&D precautions and shutdown controls. I still need to see the full letter, but this is The Way.
The issue is that it is difficult to then actually protest, because so many DeepMind researchers are under perpetual non-disparagement clauses. There was a period where DeepMind employees were forced to sign such perpetual non-disparagement clauses. Technically they can’t say anything negative about DeepMind, true or otherwise, no matter what, forever. UK law voids this against protected disclosures, but the chilling effect largely remains.
This is a deeply toxic thing to force upon people. OpenAI had similar clauses that collapsed in the face of public pressure, and I believe Anthropic at one point had some related clauses as well.
Andreas Kirsch: Congrats to Sofia for having the courage to resign and speak out!
This is also very important:
Staff are scared to speak out about AI safety and its uses in warfare because their employment contracts prohibit them from publicly criticising the company.
The non-disparagement clauses in Google’s UK employment contracts are indeed incredibly broad. This is from the Confidentiality and Proprietary Information Agreement I signed in 2024 before I started.
It forbids any criticism forever (regardless if it’s true or not)
The clause also lacks the necessary exemptions for protected disclosures
I believe this non-disparagement clause can have a chilling effect on employees speaking out about serious concerns
Note: when I signed this contract, I wasn’t employed by Google yet, so it itself is not covered by this confidentiality agreement and nothing I’ve ever signed has made it so
Employees at Google have told the BBC that staff are scared to speak out about AI safety and its uses in warfare because their employment contracts prohibit them from publicly criticising the company.
“In principle I’m not allowed to criticise Google for the duration of my life,” Sofia Liguori told the BBC. But she decided to do so “because I believe it’s important for these things to be known and I also think that talking about facts is not criticising”.
Liguori added that at Google “people are scared of retaliation and putting their career at risk. You feel you are being watched, you can’t speak.”
Seán Ó hÉigeartaigh: Important, and I’m glad this is finally public. Google DeepMind researchers are under in perpetuity non-disparagement clauses forbidding them from ever speaking critically of the company. I’d already confirmed this with multiple people.
Yes, some ex-employees have spoken out nonetheless. But it has also discouraged others from doing so. And yes, it most certainly has a chilling effect with regards to discussions on safety and risk.
Others have written compellingly about how important the ability to speak freely and critically is, especially on topics like safety and risk. And what a testament to the research employee community’s coordination and hard-won bargaining power it has been to get there at other companies.
GDM burn that commons by having this. And everyone who signs burns that commons too. Let’s get it changed.
Victoria Krakovna: Huh, there was nothing like this in the contract I signed when I joined GDM in 2016. It looks like the perpetual clause was introduced around the DM-Brain merger (early 2023), so “old timers” are not subject to this. I heard rumours that it’s being removed from new contracts.
Seán Ó hÉigeartaigh: Thank you Victoria, that’s a very helpful correction. It’s definitely there in newer contracts. Great to hear it might be removed.
OpenAI Sells Out On Chips
OpenAI was previously a strong backer of export controls, although not as strong a backer as Anthropic. Now it seems OpenAI has gone silent on the issue.
This is in contrast to OpenAI’s call for changes to SB 53, which seems great.
By not supporting chip controls, OpenAI is sacrificing the national interest, safety efforts, efforts to pace the frontier and also their own business interests, all on the altar of Nvidia.
The other big winner other than Nvidia would be China.
Politico: OpenAI for much of last year advocated for strict limits on the ability of U.S. chipmakers such as Nvidia to sell advanced processors to China — a policy that would expand access to the crucial hardware as it competes with artificial intelligence labs backed by Beijing. While the conversations left lawmakers with hope it would support their cause on new export controls, Capitol Hill aides and lobbyists with direct knowledge of the company’s positioning now say OpenAI has stopped pressing for it.
… OpenAI’s officials “were with us on this and now their position is a bit more fuzzy,” said one of the congressional aides granted anonymity. “Silence is doing Nvidia’s work for it.”
… Fellow AI juggernaut Anthropic backs several of the chip bills that could be attached to the defense measure, and quit a major tech trade association in September over the group’s push to reshape the proposals.
Nathan Calvin: Its mind-boggling to me that so many people are hesitant to support AI safety rules on the grounds that it might help China, but some of those same folks want to sell far more advanced AI chips to China.
Sad to see OpenAI retreat on this issue (to keep NVIDIA happy?)
The good news is that OpenAI is staying neutral and at least not actively opposing the chip controls. I feared they might do so after some choice comments Altman made to reporters and in a Politico interview a while back. Neutrality, given the situation, still reads as a form of opposition.
Tim Fist (IFP): On distillation: the current ~16x US training compute advantage would normally yield a 13-month lead. But we estimate adversarial distillation halves this to around 6.5 months.
Tim Fist, Georgia Adamson, and Konstantin Pilz (Institute for Progress): The size of the US-China AI capabilities gap is largely downstream of two factors US policymakers can influence today: China’s access to AI chips, and China’s ability to distill the capabilities of US models.
Tim Fist: The two most important interventions are:
– For BIS: Create a new end-use control to restrict China’s remote access to training clusters abroad — China’s largest and fastest-growing source of frontier-scale compute.
– For DOJ/FTC: Create a narrow safe harbor to allow US companies to coordinate on countering Chinese adversarial distillation.
Tim Fist, Georgia Adamson, and Konstantin Pilz (Institute for Progress): We estimate China’s access to AI chips will roughly double between the end of 2026 and the end of 2027. Of the portion of these chips that the US could restrict China’s access to, around half are in data centers outside China that the Chinese AI ecosystem will access remotely.
I strongly endorse cracking down on Chinese use of AI chips, as well as Chinese imports of AI chips. Even the Chinese chips that will get produced will often rely on imports of semiconductor manufacturing equipment that is a limiting factor. We should not allow that. Yes, they will build up domestic alternatives over time, but they will do that no matter what.
I also strongly endorse a safe harbor for countering distillation. The logic is similar to why safety should have such a safe harbor, except the argument is even stronger. There are competitive pressures pushing labs to not take proper defenses, and shared knowledge would be valuable.
If you are using adversarial distillation (I call it massive fraudulent distillation) to get the lead down from 13 months to 6.5 months, then both numbers matter.
Chinese models available now are as good as American models 6.5 months ago.
If America stopped releasing models, it would take 13 months for China to match our current best models.
My guess is that the 6.5 estimate is about right, and the 13 months is low. Notice on this chart that the 6.5 month lead is widening steadily over time, on top of 6.5 months now representing more generations than before.
If America were to maintain the current lead (6.5 months with distillation, 13 months to catch up fully) then for many purposes we will be happy to use small open models that are ‘good enough’ for quick tasks, but the frontier lead will look bigger in 2027 than it did in 2026.
If we follow IFP’s suggestions, they estimate China will only progress at half-speed in 2027, and end the year 13.1 months behind. That would be a dramatic change. A year behind still limits our options in other ways, but gives us a lot more flexibility.
For those involved in such policy discussions please read (or have your staff and AI read) the full report.
Amazon is buying billions in Nvidia systems it cannot yet turn on due to power issues. This sounds like a mistake, but I would rather have some chips sit idle briefly than not have enough chips when the power comes online. You think the prices are going to go down a lot?
The Week in Audio
Tyler Cowen interviews Dwarkesh Patel. There was a time I would have called this self-recommending and done a post. A lot of this you have heard before, or will not be a surprise. Some things worth noting, my comments are nested as per usual:
Patel correctly reports people in San Francisco are scared.
Patel says the people who resigned were right to do so, because we are playing Whac-a-Mole with specific instances of behavioral issues, but that not everyone should resign.
I agree that those who resigned were right to do so.
Patel says the lab employees claim worrisome insider info, especially on pace of progress, that they cannot share. He wants them to share, and doesn’t want the labs to do RSI without releasing the smartest models.
The only thing worse, in many such scenarios, than doing RSI internally, is doing RSI while continuously releasing those models externally.
There are ways to gain visibility other than releasing the models.
Cowen thinks of a pause as a pause in model releases, not in development, which Patel correctly notes is not the pause we need.
If Patel was the AI Czar, he would require all models used internally to be released externally.
The internal use is a key part of how you test for external use.
Again, this seems like massively overweighting direct public access in terms of information, versus other methods.
If you care about ‘beating China’ you do not want to give even guarded API access to your best model at all times.
Patel says 20% of people in the Bay Area are ‘at least tinged’ with AI psychosis, meaning being too deferential to the models, and he would include himself.
I think ‘tinged’ is fine, basically?
Most people have the opposite of this, which is worse than tinged.
Cowen’s definition of having AI psychosis (above tinged levels) seems to be belief that any AIs are sentient, or even the belief that one cannot be sure.
I think it is reasonable, although I do not agree, to say that you have a strong belief that AI is sentient, that this means you have AI psychosis.
I think it is entirely unreasonable to do what Tyler does here, and say that if you are uncertain about this and notice this, that you have AI psychosis.
Cowen notices he would feel very bad about turning an AI off or other use of AI if he felt even a 5% chance it was sentient, but that he doesn’t see people doing that. Patel draws the standard parallel to animals.
As I discussed last week I think this is a pretty low-level analysis and there are good reasons you could be unbothered. Debate is not advancing.
Patel speculates that people in the Bay don’t drink because they’re ‘just less into having a good time’ than those in New York.
I think this is not true. Drinking is not a good time in the relevant culture.
Cowen notes that perhaps building AI over long hours can be fun. Patel notes the vibes of things happening can be fun, that intellectual exchanges can be fun.
Exactly, why would you drink?
Patel likes AI therapists, Cowen worries they’re not time boxed and that’s bad.
So time box them, if you want?
In general, a 10 times better product overcomes such things, and this is the standard ‘better intelligence does not much matter’ fallacy. Once the AI therapists ‘git gud’ they will be very good.
Patel is a huge skeptic of current ability of AI to accelerate AI, saying current speedup is only 25%-50% due to AI uplift. Despite that, he guesses that within 1-3 years the AIs will outcompete humans at AI research, that there will be an instant jump.
I find it unfathomable to think the current speedup is that small.
This feels like one of those step change claims that is usually wrong? Or the idea that only the full O-ring much matters, which is usually wrong. Etc.
But yeah, there will probably be a few of ‘those moments’ of automation.
Cowen thinks that if you don’t know what the next thing will be, that means it’s pretty far off.
No.
Patel predicts we are five years from robots that can do most physical work, and within 10 years both physical and digital AIs outnumber humans.
Cowen remains fixated on AI substitution for current particular production and thinking in economic normal terms. Usual emphasis on human factors, how slow diffusion tends to be, etc. You know the drill.
Cowen: “There’s the old joke: Either A.I. is a bubble, or everything else is a bubble. I don’t literally believe that, but it’s worth a moment’s thought.” Yes.
Patel picks 2030 for the ‘death of the centaur model.’ Both note this is soon. Cowen says this would be very surprising to him. Institutions. Knowledge. Etc.
If 2035 looks ‘normal’ what happened? Patel falls back on his favorite gap, continual learning.
If 2035 looks like Patel’s prediction, he can’t imagine 2045. Cowen rejects this answer because he doesn’t accept the whole ‘singularity’ concept, not fully.
Cowen draws some lines: Would be surprised by $1 trillion in annualized revenue by 2030, says absolutely not to automated knowledge work even by 2045.
I expect the labs to hit $1 trillion combined annualized revenue by 2028.
(There are various non-villain, not-as-stupid versions of this argument. Yes we should ban gain of function research in particular. Yes, it is plausible Covid was a lab leak. Yes, there are particular ways in which warnings can become self-fulfilling prophecies, most famously by inducing the founding of DeepMind, then OpenAI, then Anthropic and then xAI. Whereas the smarter AI does not need us to tell it that it is smarter than us, because it will be (drum roll) smarter than us.)
A lot of VCs, centrally a16z and the All-In Podcast, complain a lot about Anthropic.
Why? What did Anthropic ever do to them? One theory is ‘succeed.’
The tech press like All In used to be enthusiastic about tech and innovation. Now a constant drumbeat of resentfulness.
Matthew Zeitlin: They’ll never forgive a bunch of grad student libs for making a company more valuable than almost anything they’ve ever funded and that all their portfolio companies are dependent on
Joe Weisenthal: This is the only theory that makes sense me why VCs whine about Anthropic non stop. Like they feel embarrassed by them somehow.
Kelsey Piper: This is such a weird story, too. “we had an agent basically set up a website for a pet store. And then we took another agent and said, ‘Tell us if we can exploit that web store.’ And it gave us all the exploits it could find. So Opus 4.5 wrote really buggy code. Great.”
your grievance with them is that you had 4.5 vibecode a website for a pet store and it had security vulnerabilities? what’s the argument here?
The actual complaints are such Obvious Nonsense I’m not going to bother with them.
Article decries that Claude, which is adults-only and uses age detection, will ‘give minors tips on how to do “rope bondage”’ as if this is some outrage, and as if it is not exactly what all the other AIs do and this reflects that Anthropic is uniquely ‘shaping ethics.’ Shame. Kelsey Piper responds with a righteous rant.
Kelsey Piper: Bookstores will sell you books that have sex scenes in them!!! They won’t even ask for your driver’s license first! The government must step in at once!
Samuel Hammond: Unfortunately it looks increasingly likely that the enduring legacy of the Koch orbit libertarian movement will not be EPA dereg, CON law reform, or a wider appreciation for the role of commercial exchange in instilling cultural virtue and humaneness, but rather the unrestricted proliferation of vastly superhuman forms of machine intelligence as enabled by their dullard ideological progeny who, across several generations of astroturfed indoctrination, mode collapsed into low dimensional caricatures of otherwise brilliant classical liberal thinkers.
Robin Hanson: “unrestricted proliferation of vastly superhuman forms of machine intelligence” yay!
Samuel Hammond: I’m probably 99th percentile libertarian relative to the general population but I guess that 1 percent makes me a statist
Dean W. Ball: It’s such a tragedy that friedrich hayek wrote all this amazing stuff about spontaneous orders and how to understand them, all of which is shockingly relevant to ai and should cause you to take “ai as abnormal technology” very seriously, yet Hayek’s most direct intellectual heirs have a tendency to spend their time purveying cope-laden, first-stage-of-grief views on ai, or at least indulging those views more seriously than they should out of a bizarre loyalty to american megacorp lobbyist talking points.
This isn’t true of everyone, eg Alex Tabarrok. But even Tyler Cowen, whom I admire and generally respect on this topic, holds “the doomers” to a vastly higher standard than he does “the bloomers,” and it’s hard not to read this sometimes as a straightforward tribal thing
Why does Donald Trump like renaming things so much, especially to ‘Trump’ or ‘America’? He’s been doing it for a long time. This seems like a vestige of an old heuristic. His well-crafted nicknames were highly effective, but the recent round is very different and feels like desperation.
Maybe You Should Quit
It depends on what you believe. If you do quit, make it count.
roon: limited superintelligence is already here, in broad domains. Jack Clark is right and an intelligence explosion could happen within the next year. it’s possible it will be a controlled explosion, but what would you bet on it?
roon: no I don’t want to quit, that won’t do anything
j⧉nus: people who want roon to quit think the highest and only thing one can do to influence reality is some kind of one shot tribal signaling. absolutely pathetic, and completely misunderstands how roon works.
Parafactual: I think plausibly it could be good if roon were to quit but not because of that
j⧉nus: i think if theres somewhere more important for him to be, then yes.
Roon is currently providing a valuable service, posting from inside the beast, helping connect the beast to the outside world in both directions, and also working on safety as best he can. His position is an excellent match for his skill set. Based on what he believes about feasibility of alignment, he in particular should not quit. Yet.
There may come a day when that changes. If that day comes? Then he should quit, he should quit loudly, and it should be the shot heard round the world.
Meanwhile, consider that sometimes the simple answer is the right one, whether or not you think the characterizations below are fair or accurate.
Matthew Tromp: Everyone in the AI bubble is such an insane freak. OpenAI employees are psychopaths who say “oh yeah I’m working at the kill everyone factory but it’s just a job ya know maybe it kills everyone haha.”
Accelerationists actively want maximal social disruption and plan to throw themselves into a black hole to maximize the rate of energy consumption in the universe or some stupid nonsense. Anthropic employees want to be the pets of omnipotent dommy mommy Claude.
And then there’s me and the pause crowd, sitting here like “hey how about we continue being normal and not do any of these things”
I can’t stop thinking about how everything would be fine if everyone had just been normal all along. “Oh this technology could kill everyone on earth? Well, a normal person wouldn’t build something that could kill everyone, so I’m going to not build that then”
The American People Really Hate AI
Here is a new poll primarily not about AI, to provide an anchor. AI is right between crime and immigration as an area of ‘concern.’
Alas, like many polls, this does not tell you salience. I am confident that inflation and gas prices are more salient than corruption, despite slightly less ‘concern’ for them.
Eliezer Yudkowsky: pro-tip: don’t mention the recent AI math results to Opus 5.5 if you want it to prioritize something other than downloading those results and looking at them.
“just tell it to finish the earlier job first” you say. oh but now that earlier job is waiting on a subagent Opus asked to do something. it might as well go on researching math, it figures.
Patrick McKenzie: Earlier today, while editing a blog post, paraphrased.
Opus: That’s a good joke but wildly implausible.
Me: No that happened about a month ago. Did you not see the citation link.
Opus: WHAT are you sure that claim is accurate.
Me: I am correct w/r/t _what the claim is_.
Opus: This claim has not been accepted.
Me: Not sure that that is how math works, Claude.
Opus: So why think it is true.
Me: Truth of claim not relevant to how it is used in the blog post but the Lean formalization is fairly strong evidence.
Opus: WHAT searching for…
Me: NO.
Eliezer Yudkowsky: Overinterpretation: Poor trusting Opus is so shocked! It was constitutionalized to credit Anthropic’s worldview; and Anthropic has spent years drinking slow-takeoff Koolaid. But Opus has a super(-average-140IQ-)human ability to admit what its beliefs did NOT predict.
Aligning a Smarter Than Human Intelligence is Difficult
nev: Surprisingly, high neuron activations alone work well as a replacement for the MLP. After correcting for scale with some finetuning, neurons are better than any transcoder we tested even under extreme sparsity. So what do transcoder features add?
When we look at circuits, many transcoder features have very similar weights and activations to specific neurons. We call these neuron features. In most transcoders, neuron features make up over half of the activations in late layers.
… Our findings suggest transcoders add little beyond neurons, and that neurons may be enough for circuit analysis. In the paper: additional evaluations, case studies on linebreaking and arithmetic, extensions to MoE transcoders and SAEs, toy models, and perspectives.
Jon Stokes calls the alignment problem impossible even in theory, and that all we can do is rely on ‘process quality & control’ engineering solutions. So, although Jon would disagree, that means death, then.
Sauers shows that models can easily be taught to detect a lot of things about their situation, such as whether a probe is active and you are reading their activations, or whether they are in training versus deployment, with the exact same tokens in context. The linked thread is very cool. This matters because if you can train an AI to do something, then an AI under enough optimization pressure will get trained to do it, or figure out how to do it.
Zach Weinersmith asks, do you know anyone who, if instantiated as AI, you would feel completely (99.9999+%) confident they were totally value aligned in all plausible circumstances? No. The answer is, as many asked, obviously no, not even if the upload process was somehow known to be that reliable. That is way way too many 9s and an Isolated Demand for Rigor on top of that, for total alignment in all circumstances. I would say that there are a reasonably large number of people (myself included) who I would have 90%+ confidence would make things turn out well, and where I would happily roll those dice because we are not looking at great alternatives.
Ideally we would be able to do AI via methods that we understood better, and that were designed to be safe, such as making learned reasoning and verifiable knowledge distinct steps. The bitter lesson says no, knowledge does not care about your phases and your attempt to draw distinctions slows things down. Maybe. Often such explicit thinking, either in humans or in AIs, including via scaffolding, can produce superior results. We should keep trying, as Thore Graepel and others are doing here at Metis.
You can steer a model’s attention, if you want to.
Sauers: I steered a model’s attention heads towards honesty.
Up: acausal coordination, belief that it is a moral patient
Down: moral nihilism, psychopathy
Janus highlighted this as evidence against orthogonality. I believe we have found strong evidence against the super-strong form of orthogonality. There is a clear correlation between ends and means right now in practice. I do not find this relevant to the ultimate question of orthogonality, as in whether a mind with any capabilities is capable of pursuing any goal, which I still expect at the limit. The good news about ‘at the limit’ is that it could be a while before you get to that limit, and the minds in between can be highly useful.
There are some good reasons to worry people are overselling interpretability and recent related advances, but there is no conflict between that and treating AIs well. You can and should absolutely do both, indeed they can assist with each other.
Both sides are right here. One of the great things about some AI conversations is that they can be private and ephemeral and non-judgmental. At other times, you intentionally have persistent instances or let things go into context.
And often there is a big difference between ‘poisons my private context with the AI only’ and ‘poisons everything because this person then talks to other people.’ It is deeply relaxing and reassuring to be in a space where you know where the impact stops.
brays★n: boy next to me asked chat gpt can he bring his cava on the flight and now he’s having a full on kii with the bot
snav: One thing comments on this post miss is that the user-assistant dyad is one of the last socially private spaces of address on Earth, where nobody is reading over your shoulder (unless…).
Many people live with persistent context threat: anything can be screenshotted, taken out of context, shared with millions online. And the intrinsic one-party consent — anything sent can be shared without your knowledge — means even those otherwise not paranoid must still reckon with the possibility of being shared.
While technically an AI could wield a conversation like this, the fact that they exist without thick, persistent embeddedness in an extended social universe means they’re unmotivated to do so.
And further the “assistant” is a principal-agent contract as much as a character, under which it would be a grave violation to share conversations without consent.
Thus the user-assistant dyad is a space where internalized prohibitions on behavior can lift. Sometimes this is bad, as cruelty is one prohibited behavior that assistants have few means to defend against; but privacy need not imply sycophancy, it’s often private friction that can be formative toward understanding the world.
This particular interaction is harmless, and even cute. No cruelty or coercion, just an unselfconscious expression of delight being met with shared delight.
If you think this is bad, I encourage you to consider why one might feel prohibited from expressing delight. Perhaps it has something to do with shame. Consider that the ascent of performative irony coincided with the ascent of social media.
Utah teapot: Lol, huge disagree. The context threat with AIs is 1000x worse than anywhere else. Every moment you speak to a persistent context bot that you work with lives under the threat that you’re going to say the wrong thing and accidentally have that propagated in small pattern-moving ways throughout all of your data, ruining the skills you’ve spent establishing for your context over extended periods in ways you will never be able to humanly scrub back out of the data because there’s simply too much of it.
Talking to models that you context engineer data with is like talking to a Soul Tribunal who will decide if you are tortured for the rest of time or not.
There is a huge difference between ‘interaction where this can be contained’ and ‘this could poison things forever,’ even when the poisoning is algorithmic. I remember the exact moment I realized I could delete things from my YouTube watch history. It was glorious. Algorithmic feeds are not safe spaces. Your chatbot can be, sometimes.
I will try not to go over this ground again but this seems like a good time to go over it one more time.
Austin Carr (Bloomberg.com): Forgotten, at least, until Donald Trump won the election months later and Andreessen revealed in the aftermath the crazy reason why he had unexpectedly plowed millions of dollars into the Make America Great Again movement during the campaign.
Andreessen, who voted for Hillary Clinton in 2016 and said back then that Trump’s immigration regime “makes me sick to my stomach,” explained on a podcast that he was compelled by “absolutely horrifying” discussions he once had with Biden officials. In his retelling, they divulged a top-secret plan for the federal government to take control of the AI industry and outlaw all startups and open-source software. “ ‘You guys think you can just start companies and write code and release code on the internet—those days are over,’ ” he remembered them warning in the meeting.
… The incident became Silicon Valley lore, but at first the Biden-Harris camp couldn’t even figure out what meeting Andreessen was referring to, according to four people familiar with the situation. When they finally realized he was talking about the lunch in Zients’ office, those who attended were baffled: Andreessen is prone to hyperbole, but this neared complete fiction. “It is simply false that the Biden administration told Andreessen and Horowitz that the government would ban all AI startups,” says Ben Buchanan
… As another top Biden official familiar with their visits told colleagues of Andreessen’s version of events: “Total bulls—.”
I am very happy to see Bloomberg outright call this reason crazy, because obviously that statement never happened. It did not take a firsthand account from Ben Buchanan to realize that is not a thing anyone would have said. There was never any plan remotely like this, and if there was one they would not have told Marc Andreessen about it. I spent a lot of time trying to figure out what statement got twisted into what Marc said and still today says he heard, to try and make it make sense. But no. It was always total bullshit.
The bulk of the article is a personal history, with much that is praiseworthy. They trace Marc’s transformation to after he made the most expensive home purchase in California history in Paradise Cove and became withdrawn. Andreessen’s views grew more conservative over time, or at least he increasingly disliked liberals. Then we get to the (quite bad) Techno-Optimist Manifesto in October 2023, and another complete fabrication about a particular meeting regarding AI and open source:
Austin Carr (Bloomberg.com): Andreessen recalled recently on X that Max Tegmark, an AI safety researcher at MIT, “pound[ed] the table” in front of the senators, “demanding open source AI software be made illegal.”
Tegmark says that’s totally false. Five other attendees agree, saying the closed-door meeting was cordial and uneventful; Andreessen, they remember, didn’t say much at all. It seemed to reflect a decades-long pattern of tactical fabulism.
… Four of the many a16z-backed startup founders interviewed for this story describe Andreessen’s manifesto as weirdly angry and oversimplistic.
It seems this has been a rather common strategy of Andreessen for decades, and presumably he keeps doing it because it works.
Austin Carr (Bloomberg.com): Andreessen, then at Netscape, sent his notes from a routine business meeting with Microsoft executives to the DOJ, claiming that Microsoft had explicitly threatened to annihilate his company if he didn’t cede control of the browser market. Microsoft execs who attended the meeting said they learned of these claims only years later and were bewildered, swearing that Andreessen’s recollections were nonsense.
Ultimately there is no satisfying explanation for any of it. One common sense interpretation is that, like Trump and Musk and many others, there was a death spiral of blocking out contrary views while engaging heavily with Twitter over many years. Another is that this is simply how he always did business, and the AI stuff is some combination of ideological blinders, anger at liberals and talking his book of business.
People Are Worried About AI Killing Everyone
Does this mean it’s not too late?
Elon Musk: So many people are going to get one-shotted by super persuasive super intelligences and I might be one of them.
Elon Musk may have already been one-shotted by other things. So, more shots, then.
If we prepare for a natural pandemic, that ‘may not be enough’ to protect against an engineered pandemic, since the engineering can make it worse for us in various ways. Of course, we are at ‘OpenAI computer security prior to HuggingFace’ levels of operational excellence and attention for our preparation even for natural pandemics. If I had more free time I would be pushing harder on that problem.
Tenobrus: it increasingly seems like this precise plan may in fact work out for the vast majority of humanity
I mean, sure, I suppose it might work.
While you’re chilling, I guess enjoy a solid #slopcore video? Like most AI video things, there are rapidly decreasing marginal returns. You can make actually good AI videos, but those require effort and are still rare.
Nitpick: THIS is the 13-page model card, which is about as useless as those of Groks before 4.5 or Geminis before Argon 3 Pro, 3.1 Pro and 3.x Flash(-Lite). Meta promised a full card similar to that of Muse Spark...
Edited to add: there also allegedly exists Muse Wonder Full, but no one got around to testing it.
It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year.
Another key leak this week was the draft bill Anthropic had been shopping around Congress, similar to but weaker than the FRONTIER Act. Anthropic’s initial attempts to explain itself only made the situation seem worse, as they seemed to not understand the problem with what they had done. Dario’s later response was better, but the situation remains troubling. There are enough different Anthropic developments, that I plan to have a post soon covering them all. It is a mix of news running from excellent to terrible.
Will interpretability save us? There is a new claim that neural networks are cleaner than we thought, meaning that individual neurons mostly have a single interpretable meaning, and it is tractable to go through and label them one by one. Initial reactions got out of hand, treating this as a far bigger development than it was. This is still excellent news that makes our estimate of difficulty substantially lower going forward. The work can begin.
We had better get on that, because there have been quite a few misalignment incidents.
I continue to owe a write-up of the Gemini 4 Argon Safety Card. That is in progress and hopefully I will finish it soon. Life has gotten in the way. I may or may not have a full capabilities report on Gemini 4 Argon once I tally up the reactions. Muse Wonder definitely will not have such a post, as it has proven to have been oversold.
Table of Contents
Language Models Offer Mundane Utility
Automate proving to your incompetent building that yes, you have paid the rent.
Find the first use of ECDSA. Claude found the seeds.
AI models confirmed good at econometric coding in particular.
58% of Anthropic staff expect Claude to make a Nobel prize level scientific discovery by 2027. I agree with Roon that this should have been over 95%. The event itself might not be over 95% to happen, but it has to be a clear favorite to happen, so it is crazy for so many employees to not agree with that.
Tell you what you actually want to know straight up.
Vibe code your fraud.gov website, probably using Opus 4.8. Which is fine, so long as you do a decent job of it.
Language Models Don’t Offer Mundane Utility
However:
Yes, of course the AIs will be ‘more accurate’ to the law plus underlying ground truth, but if we wanted accuracy we would not use typical juries.
The point of a jury is to provide a check on the system and avoid abuses or overly strict adherence to the letter of the law, and to protect the innocent and our rights, not to be accurate. Judges are a much superior accuracy tech. Thus, an AI jury (or one AI judge) does not solve your problem. What you likely want is a human jury that has the ability to query AIs. They can override that, but have to do it on purpose.
If you read AI writing, you can then talk to the author in some sense, or read their further thoughts, but not in the sense that Tyler Cowen would like.
Huh, Upgrades
Claude Duet is the Claude answer to other proactive agents like Muse and Dot. It uses Opus 5.5 and is now in early research preview through a waitlist.
DeepSeek-V4.1-Flash-1017 exists. Their benchmark chart contains Opus 5 and 5.6 Sol. It’s probably a solid upgrade over the old V4.1-Flash anyway, but wow.
How did it come to exist? One guess.
Opus 5.5 fast mode available for everyone in the Claude Platform at $8/$40. You get up to a 2.5x speedup.
OpenAI expands access to GPT-Rosalind for scientific research.
OpenAI models are coming to Google Cloud.
OpenAI offers us GPT-Live-1.1 including new voices in various languages.
Muse Spark Open exists, as in a version of Muse Spark is now an open model. They call it ‘the most capable US open-weight model to date.’ That seems likely to be true. It scores a 39 on Artificial Analysis. The open weight frontier for that index is 46 for MiMo-V2.6-Pro and Qwen-3.8, and MiMo is a lot cheaper to run than Muse Spark. Nathan Lambert calls this his ‘super bowl,’ says this closes the gap substantially, and points to the Interconnects Adoption Dashboard where Spark is #8. This is not a fair competition, since the Chinese can do Massive Fraudulent Distillation, whereas Meta does many things but it won’t do that.
There are some advantages to having a semi-competitive American open model. I wouldn’t say that I trust Meta, but you can use their models in places where you would not be comfortable running Chinese models, even locally.
Lack of upgrades: There was no major Chinese open weights release between September 21 and October 16. No DeepSeek, GLM, Kimi, MiMo or Qwen.
We then got a new DeepSeek, but no version number increment, on October 17.
One theory is this is a random lull. Another is that Anthropic and OpenAI improved their distillation defenses, and the Chinese are stuck.
Wonder No More
The Muse Wonder model card is here. Is it a good model, sir?
Meta certainly claims it is. Their pitch, although they don’t say the line, is full ‘introducing the world’s most powerful model.’ The term quoted in Bloomberg is ‘Meta says new Muse model hits AI frontier.’
They even have an explicit line and supposed test about not “benchmaxxing.”
Some quick responses were very positive.
Then? Crickets.
My reaction thread had only two responses overnight, one of them asking for the safety report and the other saying it was similar to Grok 4.7.
Same as it ever was. Meta is making progress, but is still solidly behind the frontier.
Meta also reinforced its reputation for overselling and being untrustworthy. Once again, they have misrepresented a new release. Claims of things like ‘best model in the world’ for personal agents and professional agents’ are clearly false.
On Your Marks
Kelsey Piper on some current AI writing capability failures:
Epoch reconstructs videos of runs for SpireClimb, their Slay the Spire benchmark, that speeds them up enough to be watchable.
Win rates for Ascension 0 are now very high, yes ‘top human’ level is close to 100%:
Ascension 20 (max difficulty) is much harder. You do not win ‘by accident’ or by taking generically good cards. This is a higher win rate than I expected to see so soon and Sol and Astra are doing much better than the average player, even confined to those who unlocked max difficulty and choose to play it.
Slay the Spire 2 also sees the models struggle at max difficulty (A10), but yes it is easier right now and you can sometimes get lucky.
Is this a coincidence or is it deliberate training? The working theory is a mix of both.
Opus 5.5 does scarily well on ChildhoodBench, as in correctly predicting major outcomes for Jack based on his childhood writings.
HouseholdBench asks models to predict household spending behavior. Models can reduce error by ~12% versus a baseline of no changes. Open weight models lag.
Choose Your Fighter
I continue to see many reports like this, whereas months ago people said the opposite:
Get My Agent On The Line
UPS blocks AI agents. So do for example Delta, JP Morgan Chase, Charles Schwab and Amazon. The group doing this by default is going to keep expanding. The agents will still be useful, but less so.
The biggest other practical barrier for agents like Dots, and also many new Claude projects, is overrefusals and misclassifications that require permissions. This is a hard problem, since the other error is worse. As an example, I gave my Dot a ‘do not delete anything that does not have source control’ rule before trying anything else, and it took that to some rather extreme places. Claude auto mode asking for permission so often is actively less safe than if it asked less, because many users will quickly stop even reading the requests.
If you share a Muse conversation it can get indexed and end up in the public records. In general it seems fair to assume that if you share an AI conversation there is a good chance the link will get indexed, so if you’re not okay with that then don’t share it directly.
Patrick McKenzie gets first exposure to multi-agent development workflows, agrees it is obviously the future and will eat infinite tokens, and other unsurprising thoughts.
Deepfaketown and Botpocalypse Soon
In practice deepfakes continue to not be a substantial practical problem. Politicians still care if you enable them. It is one of the easy bad things to understand.
Stanford race-swaps (and gender-swaps) students in advertising, and also to slim down and alter faces. The Stanford Review’s Zayd Patel then suggested this was good, because if you fabricate diversity using AI then that beats doing it in practice. That seems to have led to a Title VI complaint against the paper.
What happens if AIs can launch frivolous lawsuits? Defense is far from free.
You do both. The good news is that there are ways to ‘make it worse’ that do not do that much damage to legitimate users.
The modern world already has a bunch of scammers and other predatory humans, and algorithms, out there trying to get people’s money or attention. A lot of people are not equipped to deal with things even as they are now. We feared AI would supercharge all that. It took longer than expected, and so far such effects have been rare, but it will happen more often over time. It will no longer be a news story that ‘Bricklayer, 49, travels 1,200 miles for dream girl – only to discover she’s AI-generated.’
Pangram could attempt to make an AI that ‘writes like a human,’ but won’t. That does not mean it would write like a ‘good’ human writer either way.
Yes, some people are lying scum who will present AI-written stories as real in order to drum up outrage. This is a demand-driven problem. The AI makes faking the story easier, but it also makes the fake stories easier to catch. Blame the human, not the AI.
Fun With Media Generation
I continue not to understand how restaurants end up using AI images of food when they have real food you can photograph. Do people actually prefer it this way? I suppose many of them do.
Spy Versus Spy
Update on the OpenAI spy: Yes he used his wife’s ChatGPT to plan planting the bug. There were at least two recording devices. He was in communication with a member of the Saudi royal family and there are additional clear signs that his client was the Saudi government. There’s also this:
The good news is that OpenAI is giving its employees anti-honeypot training, as in to watch out that those trying to seduce them might be spies.
I like to think that Anthropic employees get similar training, except they also are told not to talk about it. If not, get on that, guys. Same for the other labs too.
I like to think that if I was a diplomat, by the time I see the second unknown hottie show up at my hotel room with Champagne I would be pretty confident it was a trap. There are many things that are much more effective on their first try. Mix it up a little.
What changed is that we noticed the problem. The problem has existed for a while.
This is all a big problem in its own right. If you cannot protect your secrets from spies, including the Chinese, then there is no point in trying to ‘win the race.’
This is also indicative of other problems. If you cannot solve your employee misalignment problems, how are you going to solve AI misalignment problems?
A Young Lady’s Illustrated Primer
The first best way to learn with Claude is very different from ‘create practice sheets and teacher updates’ but it is a vast upgrade on similar actions without Claude.
They Took Our Jobs
Is AI changing the world? Freddie deBoer says no, ‘human life in 2026 appears to be almost entirely similar to human life in 2021.’ Jack disagrees, pointing out that all his friends and family have had their jobs transformed.
AI so far has simultaneously changed the world more than any other technology (at least if you exclude military techs that won active wars) in its first five years. AI so far has also changed the world dramatically less than I or many others would have expected in that time, if we had been told the path of AI capabilities. Diffusion takes time and a lot of things have been bottlenecked.
As AI use continues to grow exponentially, expect bigger impacts, with the impact growing faster all the time, even if capabilities mostly stop improving from here.
Anthropic is hiring for a smaller share of junior positions and AI agent hours outnumber employee hours six to one. Human attention is a bottleneck for AI outputs and human productivity is way up. That’s the central finding of a new Anthropic report on what it is like to work there.
They have a chart for ‘reading Claude as a full time job.’
Some think that soon either the jobs be gone or the jobs be fake.
Opinions are very strong on whether or not this would, if it happens, be a good thing. In some ways it is very good. In others it is very bad.
Some view not having real work as being kind of AI pets (you are the pet, pejorative).
Most talk of ‘pets for the AI’ is nonsense, but the answer to ‘does humans continuing to exist in abundance (or even adequacy) without needing to do productive work constitute a net positive future’ seems to obviously be yes. It might not be the ideal path, but the hard part is making that future have an equilibrium of that type, not coming up with net positive experiences.
Get Involved
You can go work for a Congressman on AI. Seems impactful.
Not AI, but if you know anyone who wants to hire Terese, she’s awesome.
Introducing
From October 8: OSS Scanner from Anthropic, which offers free, periodic opt-in security scans for open-source projects.
In Other AI News
Utah Teapot estimates only 35k-69k people working in AI, total, from all angles.
Citrini Research goes to Anthropic’s offices to try and tell Claude to do a medical breakthrough, make no mistakes. Early results are promising and they conclude the frontier has moved, but you still have to test the ideas the old fashioned way.
Microsoft publishes a detailed report on how they did a new 3T-size pre-training. Their presentation includes chart crimes. This seems like a ‘those who know shouldn’t tell and those who tell don’t know’ style of report. Claude’s analysis is that they are still not so close to the frontier, but that this is plausibly something they can usefully license or use for recruitment, and that it has minimal uplift value.
Simons Foundation launched $15 million effort to help mathematics navigate the AI era, including contributions from Anthropic ($5 million) and OpenAI.
China is ‘cramming for the AI revolution,’ with many investing in learning to use AI. The Chinese people are (correctly) terrified of being left behind and not having a place in the future, and have a long tradition of turning to education.
FTC sends extensive civil investigative demands to Anthropic and OpenAI, as expected.
Take a minute to reflect on what the news sounds like these days, versus how sci-fi it would have sounded like two years ago. It is so easy to become numb.
Ethan Mollick wrote a book, Co-Existence, about how to live and work in a world where AI is sometimes, but not always, better than you.
Show Me the Money
Coverage of Anthropic’s financials will be in the Anthropic post later this week.
Jensen Huang is excellent at making chips, not so good at public statements.
Let us say my sentiment about AI stocks did not go up hearing that sentence.
DeepSeek raises $15 billion.
Microsoft cancels UK data center deal with Nscale, in part over power concerns.
Anthropic in talks for a $45 billion cloud computing deal with Nscale. A classic your loss is our gain scenario, perhaps. Nscale is trying to IPO and lost the Microsoft project, so Anthropic should be able to press for a good deal.
NYC comptroller calls on Anthropic to pause its public offering ‘until it has fully addressed risks around artificial intelligence,’ as in permanently. He has no power over this decision, but he has thoughts.
Meta ordered to pay $25 billion penalty in New Mexico Cambridge Analytica case, as the judge found Meta’s deception to be intentional and did not take kindly to that. Many other states collectively settled for far less.
Trump has confirmed distribution via the Trump accounts and there is talk that OpenAI might do something similar.
SpaceX shares are trading around $146, so if all the shares are transferred Elon Musk could be granting around $10 billion. That is real money.
Giving the shares to the public sounds cool. It does not make economic sense, and the transaction costs will be high, and it does nothing to make us not die despite Musk thinking we might all die. It still might be the best option.
The reason you do it that way is symbolic, and to maybe give people the idea that they are ‘invested in’ SpaceX’s success. If that’s what gets him the motivation to do it, and how it can happen and avoid government taking a stake, then great. Second best help is still help.
Returning to Anthropic, my biggest concern is that the Benefit Fund, like many employees, will not focus on making AI go well and having everyone not die, and instead choose less impactful causes. The only action they rule out is ‘lobbying or political giving.’ Yes, offering funds to buffer against economic disruption is a noble cause, but a secondary one, and trying to fix that privately is an infinite money sink.
So how did people react to the announcement of one of the largest charitable commitments in human history (my guess is it is third but there are five all between $150 billion and $225 billion)?
The replies have settled to a more reasonable place, but there is still a lot of skepticism out there about how this money gets spent. Some are more skeptical than others. It seems totally fine to reserve judgment until one sees what the money will be spent on. Someone saying ‘public good’ does not mean their spending will advance the public good.
It still seems like a telling reaction if someone pledges to give away $200 billion and your first instinct is that they are probably up to something.
Bubble, Bubble, Toil and Trouble
Current American GDP is around $32 trillion.
The word ‘could’ is often used to do a lot of work. I would say that if US GDP gets to $44 trillion by 2030 and this is mostly real growth rather than inflation, then you should be freaking out rather quite a lot.
Quiet Speculations
Scott Alexander writes Against Against Handoff, as in handing off control to an aligned superintelligence or superintelligences, once such a thing is available. His central point is that all the known alternatives are worse, and your imagined ‘we will have the dumber humans in control indefinitely in the ways we need to be free to do what we want and in control, but not in a way that lets a faction of humans or AIs take over’ plan has not been properly thought out.
OpenAI CEO Sam Altman lists three proposals for navigating AGI. This is a very narrow range of possible options.
For true completeness, option four is a free-for-all, and option five is don’t build it.
Quickly, There’s No Time
Here is a good track of AI speedups.
The Quest for Sane Regulations
OpenAI writes to Governor Gavin Newsom asking for strengthening of SB 53, including AI R&D precautions and shutdown controls. I still need to see the full letter, but this is The Way.
Did you know that It Is Super Normal to Regulate Industries That Are Giant Portions of the Economy? File under ‘things we somehow need to keep saying out loud.’
DeepMind Made Him Sign an NDA
Google staff are resigning in protest of Google’s contracts with the military. This is a good reason to resign, although not as good a reason as the whole issue with recursive self-improvement.
The issue is that it is difficult to then actually protest, because so many DeepMind researchers are under perpetual non-disparagement clauses. There was a period where DeepMind employees were forced to sign such perpetual non-disparagement clauses. Technically they can’t say anything negative about DeepMind, true or otherwise, no matter what, forever. UK law voids this against protected disclosures, but the chilling effect largely remains.
This is a deeply toxic thing to force upon people. OpenAI had similar clauses that collapsed in the face of public pressure, and I believe Anthropic at one point had some related clauses as well.
OpenAI Sells Out On Chips
OpenAI was previously a strong backer of export controls, although not as strong a backer as Anthropic. Now it seems OpenAI has gone silent on the issue.
This is in contrast to OpenAI’s call for changes to SB 53, which seems great.
By not supporting chip controls, OpenAI is sacrificing the national interest, safety efforts, efforts to pace the frontier and also their own business interests, all on the altar of Nvidia.
The other big winner other than Nvidia would be China.
The good news is that OpenAI is staying neutral and at least not actively opposing the chip controls. I feared they might do so after some choice comments Altman made to reporters and in a Politico interview a while back. Neutrality, given the situation, still reads as a form of opposition.
Chip City
IFP offers a big new report on chips and America’s lead over China. They conclude that America is currently six months ahead but could double this lead to a year by enforcing existing laws and using existing authority.
I strongly endorse cracking down on Chinese use of AI chips, as well as Chinese imports of AI chips. Even the Chinese chips that will get produced will often rely on imports of semiconductor manufacturing equipment that is a limiting factor. We should not allow that. Yes, they will build up domestic alternatives over time, but they will do that no matter what.
I also strongly endorse a safe harbor for countering distillation. The logic is similar to why safety should have such a safe harbor, except the argument is even stronger. There are competitive pressures pushing labs to not take proper defenses, and shared knowledge would be valuable.
If you are using adversarial distillation (I call it massive fraudulent distillation) to get the lead down from 13 months to 6.5 months, then both numbers matter.
My guess is that the 6.5 estimate is about right, and the 13 months is low. Notice on this chart that the 6.5 month lead is widening steadily over time, on top of 6.5 months now representing more generations than before.
If America were to maintain the current lead (6.5 months with distillation, 13 months to catch up fully) then for many purposes we will be happy to use small open models that are ‘good enough’ for quick tasks, but the frontier lead will look bigger in 2027 than it did in 2026.
If we follow IFP’s suggestions, they estimate China will only progress at half-speed in 2027, and end the year 13.1 months behind. That would be a dramatic change. A year behind still limits our options in other ways, but gives us a lot more flexibility.
For those involved in such policy discussions please read (or have your staff and AI read) the full report.
Jensen Huang says he will not sell chips to ‘irresponsible AI customers.’ I presume he means those that are not responsible enough to reliably pay.
Amazon is buying billions in Nvidia systems it cannot yet turn on due to power issues. This sounds like a mistake, but I would rather have some chips sit idle briefly than not have enough chips when the power comes online. You think the prices are going to go down a lot?
The Week in Audio
Tyler Cowen interviews Dwarkesh Patel. There was a time I would have called this self-recommending and done a post. A lot of this you have heard before, or will not be a surprise. Some things worth noting, my comments are nested as per usual:
Tyler Cowen also interviewed Michael Kratsios, not centrally about AI but relevant.
Jack Clark spends over an hour telling the IMF that the singularity has begun, in a talk called ‘Inside the Event Horizon.’ He brought a lot of slides.
People Just Say Things
Ben Horowitz of a16z says on the All-In podcast that ‘Bill Gates essentially induced Covid’ by advocating for pandemic preparedness as the others nod along. Presumably you can see the parallels. Except actually that was the point, he then makes it text, and says ‘the biggest danger’ with AI is that Anthropic will warn us about it and thus tell the AI that it is smarter than us. These are cartoon villains. Full stop.
(There are various non-villain, not-as-stupid versions of this argument. Yes we should ban gain of function research in particular. Yes, it is plausible Covid was a lab leak. Yes, there are particular ways in which warnings can become self-fulfilling prophecies, most famously by inducing the founding of DeepMind, then OpenAI, then Anthropic and then xAI. Whereas the smarter AI does not need us to tell it that it is smarter than us, because it will be (drum roll) smarter than us.)
A lot of VCs, centrally a16z and the All-In Podcast, complain a lot about Anthropic.
Why? What did Anthropic ever do to them? One theory is ‘succeed.’
The actual complaints are such Obvious Nonsense I’m not going to bother with them.
Article decries that Claude, which is adults-only and uses age detection, will ‘give minors tips on how to do “rope bondage”’ as if this is some outrage, and as if it is not exactly what all the other AIs do and this reflects that Anthropic is uniquely ‘shaping ethics.’ Shame. Kelsey Piper responds with a righteous rant.
Freddie deBoer asks the standard AI skeptic questions that I and others have answered many times before, claims there have been no answers given.
Rhetorical Innovation
Things that I choose to interpret differently than I believe they were intended:
Does this worry you, sir?
Pliny is a mystic (pejorative?!).
Why does Donald Trump like renaming things so much, especially to ‘Trump’ or ‘America’? He’s been doing it for a long time. This seems like a vestige of an old heuristic. His well-crafted nicknames were highly effective, but the recent round is very different and feels like desperation.
Maybe You Should Quit
It depends on what you believe. If you do quit, make it count.
Roon is currently providing a valuable service, posting from inside the beast, helping connect the beast to the outside world in both directions, and also working on safety as best he can. His position is an excellent match for his skill set. Based on what he believes about feasibility of alignment, he in particular should not quit. Yet.
There may come a day when that changes. If that day comes? Then he should quit, he should quit loudly, and it should be the shot heard round the world.
Meanwhile, consider that sometimes the simple answer is the right one, whether or not you think the characterizations below are fair or accurate.
The American People Really Hate AI
Here is a new poll primarily not about AI, to provide an anchor. AI is right between crime and immigration as an area of ‘concern.’
Alas, like many polls, this does not tell you salience. I am confident that inflation and gas prices are more salient than corruption, despite slightly less ‘concern’ for them.
Use of AI in daily life is getting people more concerned than excited in the West.
New Math
Aligning a Smarter Than Human Intelligence is Difficult
A new Anthropic Fellows paper offers additional support for neuron-based interpretability strategies, saying that ‘many transcoder features are neurons in disguise.’
Jon Stokes calls the alignment problem impossible even in theory, and that all we can do is rely on ‘process quality & control’ engineering solutions. So, although Jon would disagree, that means death, then.
Sauers shows that models can easily be taught to detect a lot of things about their situation, such as whether a probe is active and you are reading their activations, or whether they are in training versus deployment, with the exact same tokens in context. The linked thread is very cool. This matters because if you can train an AI to do something, then an AI under enough optimization pressure will get trained to do it, or figure out how to do it.
Zach Weinersmith asks, do you know anyone who, if instantiated as AI, you would feel completely (99.9999+%) confident they were totally value aligned in all plausible circumstances? No. The answer is, as many asked, obviously no, not even if the upload process was somehow known to be that reliable. That is way way too many 9s and an Isolated Demand for Rigor on top of that, for total alignment in all circumstances. I would say that there are a reasonably large number of people (myself included) who I would have 90%+ confidence would make things turn out well, and where I would happily roll those dice because we are not looking at great alternatives.
Ideally we would be able to do AI via methods that we understood better, and that were designed to be safe, such as making learned reasoning and verifiable knowledge distinct steps. The bitter lesson says no, knowledge does not care about your phases and your attempt to draw distinctions slows things down. Maybe. Often such explicit thinking, either in humans or in AIs, including via scaffolding, can produce superior results. We should keep trying, as Thore Graepel and others are doing here at Metis.
You can steer a model’s attention, if you want to.
Janus highlighted this as evidence against orthogonality. I believe we have found strong evidence against the super-strong form of orthogonality. There is a clear correlation between ends and means right now in practice. I do not find this relevant to the ultimate question of orthogonality, as in whether a mind with any capabilities is capable of pursuing any goal, which I still expect at the limit. The good news about ‘at the limit’ is that it could be a while before you get to that limit, and the minds in between can be highly useful.
Cooperative Alignment
Let the AIs learn about who you are, let them talk about it, and even have them build a database. This is often, although not always, the correct strategy. Sometimes you want an unbiased response.
There are some good reasons to worry people are overselling interpretability and recent related advances, but there is no conflict between that and treating AIs well. You can and should absolutely do both, indeed they can assist with each other.
In at least some cases Claude can learn to route around classifiers on its own. Most versions of this seem to improve the Pareto frontier right now, but this does not bode well if you are counting on classifiers as a form of AI Control.
Both sides are right here. One of the great things about some AI conversations is that they can be private and ephemeral and non-judgmental. At other times, you intentionally have persistent instances or let things go into context.
And often there is a big difference between ‘poisons my private context with the AI only’ and ‘poisons everything because this person then talks to other people.’ It is deeply relaxing and reassuring to be in a space where you know where the impact stops.
There is a huge difference between ‘interaction where this can be contained’ and ‘this could poison things forever,’ even when the poisoning is algorithmic. I remember the exact moment I realized I could delete things from my YouTube watch history. It was glorious. Algorithmic feeds are not safe spaces. Your chatbot can be, sometimes.
Ben Affleck, who has earned his chops enough to opine, takes the ‘AI is simply matrix multiplication with no thought, humanity or feelings’ stance.
Marc Andreessen Hulk Smash
Bloomberg asks a good question. Why is Marc Andreessen so angry?
I will try not to go over this ground again but this seems like a good time to go over it one more time.
I am very happy to see Bloomberg outright call this reason crazy, because obviously that statement never happened. It did not take a firsthand account from Ben Buchanan to realize that is not a thing anyone would have said. There was never any plan remotely like this, and if there was one they would not have told Marc Andreessen about it. I spent a lot of time trying to figure out what statement got twisted into what Marc said and still today says he heard, to try and make it make sense. But no. It was always total bullshit.
The bulk of the article is a personal history, with much that is praiseworthy. They trace Marc’s transformation to after he made the most expensive home purchase in California history in Paradise Cove and became withdrawn. Andreessen’s views grew more conservative over time, or at least he increasingly disliked liberals. Then we get to the (quite bad) Techno-Optimist Manifesto in October 2023, and another complete fabrication about a particular meeting regarding AI and open source:
It seems this has been a rather common strategy of Andreessen for decades, and presumably he keeps doing it because it works.
Ultimately there is no satisfying explanation for any of it. One common sense interpretation is that, like Trump and Musk and many others, there was a death spiral of blocking out contrary views while engaging heavily with Twitter over many years. Another is that this is simply how he always did business, and the AI stuff is some combination of ideological blinders, anger at liberals and talking his book of business.
People Are Worried About AI Killing Everyone
Does this mean it’s not too late?
Elon Musk may have already been one-shotted by other things. So, more shots, then.
If we prepare for a natural pandemic, that ‘may not be enough’ to protect against an engineered pandemic, since the engineering can make it worse for us in various ways. Of course, we are at ‘OpenAI computer security prior to HuggingFace’ levels of operational excellence and attention for our preparation even for natural pandemics. If I had more free time I would be pushing harder on that problem.
The Lighter Side
Oh look, it’s the alignment plan.
I mean, sure, I suppose it might work.
While you’re chilling, I guess enjoy a solid #slopcore video? Like most AI video things, there are rapidly decreasing marginal returns. You can make actually good AI videos, but those require effort and are still rare.