Nate Soares (MIRI): In sci-fi books and movies, AI solving a bunch of math problems that stood open for decades would’ve been a big deal. Why isn’t the mainstream media turning this into a bunch of sensational stories?
I hate to say it, but because it is math and a lot of the public don't like math. They should care, and there should be coverage. But boiling frog syndrome and nerdiness quotient, and this means no coverage.
I think it's also that the public doesn't understand maths well enough to judge the importance of these new developments. What's most visible is the AI companies themselves pushing these results hard, which they would clearly do for marketing purposes regardless of whether it is actually sensational.
Given that the mathematics community as a whole is not freaking out (or at least the reactions are very mixed), caution is probably an appropriate response if you don't have enough experience to form your own opinion.
I think it would be different if AI solved the Riemann Hypothesis (with a positive proof rather than a counterexample), mathematicians would not shut up about it and that would gather some very wide attention.
But here is a piece of key information Nate Soares understands well and general public does not understand at all.
If it turns out to be necessary to do some radical post-Transformer paradigm breakthroughs in order to achieve ASI, then the ability of AI systems to find those breakthroughs is highly correlated with the ability of AI systems to do high-grade math research.
I think AI labs understand this quite well.
Today, AI solved not one, but NINE open problems – some 50 years old.
The nine Erdos problems discussed in the new AlphaProof paper were not newly announced. At least, the first I checked (125) was announced back in February.
Last week ended on a cliffhanger of sorts. What’s in the Executive Order coming later today? What will be in the Magnifica Humanitas?
The Executive Order was postponed indefinitely, likely cancelled entirely except for work on securing critical infrastructure. David Sacks and others intervened to kill it, and American AI policy will continue to be maximally ad hoc.
Instead, we got Illinois SB 315, which is to be signed into law. It is a variation on California’s SB 53 and New York’s RAISE Act, while adding a third party auditing requirement.
The Magnifica Humanitas was revealed early in the week. I did an extensive readthrough, and have some follow-up commentary here to clear up some things I interpreted incorrectly and add richer context. It too ignores elephants in the room, not discussing AGI or existential risk and calling on people to ignore their incentives and instead do the right thing by prioritizing common good, especially access to good jobs and an end to war.
What I now believe I centrally misunderstood is that I interpreted this as a call to action, to use law and regulation to make this happen, because obviously that is the way you would actually engineer such outcomes. The Pope, I am told, instead really does think you can just ask the people to individually choose to not follow incentives. There is a section below with more on this.
A note on post frequency: For many months, I posted five times a week, and used any lulls as an opportunity to post non-AI things or split off subtopics. I have another project right now, and AI is in a relative news lull (I know, but yes, this is a lull), so instead I am using this as an opportunity to rest a bit. I will evaluate in July whether to return to a full all-weekdays post schedule.
Table of Contents
Language Models Offer Mundane Utility
Claude Code’s auto mode is highly recommended all around.
A lawyer who previously found models unusable for work due to hallucinations comes around for GPT-5.5 Pro.
Google claims its AI agents built an operating system with a single prompt and $916. Kapoor and Narayanan investigate, and find this basically press release puffery but with a grain of truth, even if the prompt was thousands of lines long and they haven’t released a bunch of other key data.
Robin Hanson on his use of current LLMs:
AI can give you massive amounts of mediocre intelligence to do various tasks where you are bad or lack the time, to give you the chance to use your comparative advantage. Being mediocre at basically everything is a massive edge.
Judges are using an AI program called Learned Hand to distill legal motions, reach conclusions and draft tentative rulings.
Language Models Don’t Offer Mundane Utility
Anthropic continues to shut out users under 18 and says there needs to ‘be an adult in the room, a human in the loop’ and cites Haidt. I strongly disagree with this policy and would want my children freely using Claude at 13 at the latest, and this decision should at most by up to the parent by then. The good news is that it is not hard to get around such restrictions.
Berkeley law prohibits any use of AI whatsoever for any work submitted for credit, with the exception of identifying sources. Good luck with that.
Reminder: Don’t run your writing through an LLM unless you actively want AI slop. If you want help, have the LLM suggest individual changes. This includes writing issues for open source projects, the same as writing for humans.
Armin tries to tell their own AI to ignore such analysis, but it doesn’t work.
Do The Math
Google DeepMind’s AlphaProof Nexus solves 9 open Erdos problems, 44 OEIS problems, and more.
Various mathematicians respond to OpenAI’s disproof of the Unit Distance Conjecture.
Anthropic’s Levent is curious, so now we know that yes, Mythos can also solve the Unit Distance problem. Sometimes it lands on the same argument as OpenAI did, sometimes it finds a different one.
Gary Marcus does his best to be skeptical of OpenAI’s proof, without success.
What will mathematicians be for once AI is better in all areas of math? Daniel Litt suggests ‘locus of understanding’ which of course is another thing AI will be better at relatively soon. But we might insist a human do it anyway, as Daniel suggests, to try and help stave off human disempowerment.
Demis Hassabis cautions that no, solving Erdos problems is not ‘true invention’ and we are nowhere near AGI as he sees it. Contra Noah Smith, AI doing things humans cannot does not mean it can do the things that humans can, or that it is AGI. But yes, they are solving Erdos problems.
This is of course a Wrong Question, as the Erdos problems are proof of ability to solve problems and indication of future ability, not primarily mundane utility. But I asked Claude and it said median of $10k-$100k but with a long tail of $1m-$50m, with average of something like $1m-$10m.
Basically we got the first math problem that is exciting in itself rather than simply as an indicator, so of course the response is ‘well, sure, it’s fun and exciting but how much money did it just make?’
The world is indeed this asleep.
Huh, Upgrades
ChatGPT for Powerpoint. Google Slides instead, please.
Codex updates:
Antigravity makes it easier to connect to the IDE.
Claude Code gets a new plugin to identify and fix vulnerabilities as you run code.
MiMo-v2.5 API pricing now permanently reduced ‘by up to 99%,’ more typically 5-8x. Lower tier models are often remarkably cheap, if your use case can handle it.
On Your Marks
Rohit gives us BenchBench, a test asking AIs to construct new benchmarks. Only one out of six models (GPT-5.2) created a benchmark that was neither too hard, too easy or too puzzle-like or checkbox-like or brittle. There’s a lot of ways to fail at benchmarks. This has to be done manually, so they only got a few shots each.
An obvious follow-up would be to offer feedback. Opus repeatedly created benchmarks that were good but too easy. That seems fixable.
Somehow I just figured out that SecureBio, which does evals on biological capability, has its own blog. Here is the assessment of GPT-5.5.
DeepSWE is a new agentic coding benchmark. GPT-5.5 leads at 70%, GPT-5.4 at 56% and Opus 4.7 at 54% are the only other models above 32%.
Apollo Research warns that models will soon be too eval aware for black box evaluations of alignment, so we will need white box evaluations instead. They want to do things like have steerable evaluation-awareness endpoints. This is an unsurprising reaction, but it is hard to express how much despair this plan and attitude invokes.
Get My Agent On The Line
The art of going meta by proxy, no Taelin is not the first to suggest this.
Anthropic talks about how they build containment for their agentic AI systems, comparing the probability times blast radius to the reward. This is going to go great.
Deepfaketown and Botpocalypse Soon
Pangram Labs shows that last year’s Commonwealth Short Story Prize was also won by an AI-written story, and that 3 of the 5 entries this year were AI. Notice that this also shows a robustness, there are no false positives at even 1% level before 2025.
I do think this is a strong benchmark. The prize is a worthless joke in terms of identifying literature you or I would want to read, or that the world should remember, but it is clearly measuring something, and winning is a hard and anti-inductive problem. If the AIs are willing, that’s impressive.
In other ‘you deserve it’ news, the author of “Future of Truth” included more than half a dozen fake or misattributed quotes because he trusted ChatGPT, oh well:
Could ChatGPT have undermined him on purpose? Unlikely, but I wouldn’t rule it out
I mean, clicking back leads to this Tweet, and come on that is so obviously not a human voice.
The thing about AI-generated content is that we are now often at a point where it would have been in many ways fine content if a human had written it in 2022, but now it causes eye glazing among those who are used to seeing it and recognize it instantly.
I don’t see why it has to be a trick, or why anyone needs to be trying to impress. Perhaps the human simply wants to aid in their communication. That doesn’t make this a good idea, and I am in a similar place to Paul Graham. If you send me a cold email, I may read it if it is relevant to my interests, but if you had an AI write it then I will delete the moment I notice.
It’s worth noting that this is actually not so difficult in terms of direct writing:
There is a broad ‘middle range’ where you cannot be sure, but good human writing is very obviously human. There is an aliveness, a perplexity, a willingness to ‘go there’ that AI simply will not produce. Often I am very confident something is not written by AI, the same way one is often confident on reflection that one is not sleeping.
What you cannot possibly do is prove you did not use AI in other ways, such as to ask questions, check facts, search the web or finding errors. There’s flat out nothing you can do, because the writing looks exactly the same, and it is exactly the same.
It’s not AI, but it is kind of still AI slop:
Whereas if it is AI but not slop, it will ‘fool’ Pangram.
I am totally fine with Pangram being a detector for ‘standard AI writing’ and for nonstandard AI writing to get through.
This is also the right mistake to be making. What is great about Pangram is the very low false positive rate. If the test has a substantial false positive rate, you basically can’t use it, as people can say it got this one wrong, and you don’t want to risk making a mistake.
Whereas if you have a substantial false negative rate, especially if it is concentrated in ‘better’ uses of AI, then that is annoying but essentially fine.
We will be putting that to the test.
Ultimately, what are you looking for? Why do you care if an AI wrote something?
One good reason is to call out people for wasting our time and polluting the commons.
Another obvious one is school or other places where it would be cheating.
Alexander Kustov’s point is, if (as with the encyclical) AI is perhaps used to do good writing, why do you care? Are you sure you shouldn’t be filtering out the human content instead?
Copyright Confrontation
Individual researchers sued for copyright infringement, simply because they torrented 70 terabytes of pirated books and there are records confirming this.
Cyber Lack of Security
Update on Project Glasswing. They have found ‘more than 10,000’ high or critical security vulnerabilities. It’s not great that it looks like most of these are unpatched.
Daniel Stenberg reports 11 new confirmed vulnerabilities halfway through the Curl release cycle, none of which even involved Mythos. It’s a new world out there.
Aaron Levie declares Jevon’s Paradox calls this failure to patch quickly a reason human engineers will never go away. People don’t understand that the AIs will, in the future, get good at other things they are not currently good at, such as patching.
Patrick McKenzie is impressed by this AI-written blog post about supply chain attacks. The post is very obviously AI written, but also isn’t hiding this.
Overcoming Bias
If you’re reliably willing to help users find left-wing artists or Communist artists, but not right-wing artists, then yes, I’d say that means your AI is de facto biased. Example here is from ChatGPT.
Long Phan extends this, such as AI failing to criticize Islam while being happy to criticize Christianity via an identical prompt, and suggests ways to measure the bias, and train to reduce it.
This is related to, but not the same, as the tendency for models to be left-libertarian by default, and for it to be very hard to move them off of this. You can have a perspective and still avoid this type of behavior, and you can have either Sentiment or Helpfulness consistency move without the other one. Paper here, website here.
I notice that this kind of training fix risks being a cure worse than the disease in various ways, and if one proceeds I would proceed with caution.
A Young Lady’s Illustrated Primer
Assistant professor at Amherst gives a fully AI-written response when asked about use of AI in the classroom for an article about how AI is ruining the classroom because students no longer engage and struggle with the material, just offloading to AI.
The students are right. If you let students get away with AI use, that is on you and your system. It’s not that hard to detect, it’s only (sometimes) hard to prove. Then there are the students that are mad at you for being mad at them for offloading their assignments to AI, because they think they are there to buy a degree with time and money, and you (foolishly) thought they wanted to learn. That’s also on you.
All the interviews find most of the teachers deeply pessimistic. They know their old model is broken, but don’t have a replacement. They don’t know how to teach without essays. They don’t know how to test in reasonable time. Some don’t feel they can identify AI writing, despite the existence of Pangram, while others find it easy on instinct but despair on what to do about it. Then there are those who are embracing AI, but they need to spend time designing new material and methods, and steering their students away from lazy use.
Unprompted Attention
Tell Codex (or Claude Code) to look through your work and find the lowest hanging automatable workflow, automate it if worthwhile, then repeat.
They Took Our Jobs
Alex Tabarrok says AI won’t take (all) our jobs, but if it did that’s a ‘rich man’s problem’ because it would mean we are all super wealthy, and problems where the pie gets bigger are problems we can solve. I would amend that to ‘distributional problems’ instead of problems, but yes. This is true in an aggregate sense if humans collectively remain alive and in control and in control over the resources, which is exactly why I worry more about the part where we may not remain alive and in such control, but the wealth existing does not mean it gets where it needs to go or solve the other problems with lack of gainful employment.
Or as a wise man once said, yes give me mo money, but also mo money, mo problems.
Tyler Cowen offers tactics to help avoid losing your job to AI.
Cloudflare CEO lays off 20% of their workforce despite strong free cash flow and growing at 30%, explains that AI lets them cut middle managers, or ‘measurers,’ and also that they had almost a million applicants for 1,111 paid internships. But don’t worry, he says. AI won’t take all the jobs, it will be fine.
Peter Lambert and Yannick Shindler offer a new paper saying yes early- career hiring is a mess but that is because of work-from-home.
We have p=0.77 between exposure measures for WFH and GenAI exposure, because the same factors create conditions for both of them, and as they admit the time series strongly suggests GenAI rather than WFH. The result is not statistically robust, it doesn’t really make sense, even places not exposed to WFH saw falling junior hiring, and the sizing does not make sense.
My guess is that what is happening is partly, yes, that WFH does discourage early hiring and favor experienced hiring. But primarily my guess is WFH is a proxy for future AI exposure, and employers are smarter about measuring this than the direct backward looking exposure measures.
Get Involved
Periodic reminder: If you want to contact me, for pretty much any reason, I want to anti-recommend Substack messages. I don’t check them often, as the messages I get tend to be spammy and their email notifications don’t tell you the contents of the message, and also they forcibly hide most links. You will think the link works, but on my end it won’t.
You are welcome to share articles and other links of interest, including your own, so long as it is NRN (no reply necessary), and you understand I will probably not engage.
You can contact me via email or Signal/WhatsApp/text (in that order) if you can figure out how, or you can DM me on Twitter (@thezvi) or PM me on LessWrong (Zvi).
You can also leave comments. I try to have a very light touch approach to the comments. I still do. If you want to be Wrong In The Comments or call me an idiot that’s fine. It’s not quite full free speech uber alles, but it is close.
The issue is: AI-written slop comments are becoming steadily more of a problem. The rule is, if you’re going to use AI, or let your AI comment, be interesting. If I notice AI comments being boring, I will silently delete them. Do it in bulk and I will ban you.
Endorsed by Janus, probably a pretty good opportunity although I don’t know them, right now deckard is in Scotland:
80,000 Hours: The Book. Career advice, EA-style, from a moment in time.
Introducing
Qwen 3.7-Max, oh look it’s impressive Chinese benchmarks. I look forward to probably not hearing anything more about it.
In Other AI News
Yo Shavit resigns from OpenAI to move to the OpenAI Foundation. I am slightly worried that OpenAI will get somewhat hollowed out in this way, but the foundation needs the help.
ChatGPT, Claude and Gemini are reliably the top three at the iOS app store.
OpenAI foundation commits $250 million to things that are not the (main) mission:
This is net positive work, and I am happy people are doing it, but it does not address the reasons why the foundation exists, as in making AGI go well and everyone not die. I do not think that economic distribution plans do much to address this.
FAI launches a physical intelligence team, to help create a policy and regulatory landscape that would allow us to use robotics and other new tools to actually build cool and useful physical things, especially in places where we are currently facing physical limits.
Show Me the Money
Microsoft cancels its Claude Code licenses due to cost concerns. You fool. This is bad news for Anthropic, but far worse news for Microsoft. If you shift developers from Claude Code to GitHub Copilot CLI, then like those who do not provide free coffee you do not want to win, and you are not serious people.
No, owning Copilot does not change that.
Hedgie then shows this theoretical graph:
I mean, yes, obviously if revenue is constant and costs go up, that is bad for you. If you spend a bunch on AI and you don’t get more productive, that’s bad. But very obviously, that is not what is happening.
What is actually happening is that Microsoft felt that using the competitor’s product, Claude Code, was undermining their own greatly inferior offering, GitHub Copilot CLI, and also making expenses look higher and they wanted to fool investors.
Show Me The Compute
Anthropic’s total secured compute, a lot of which won’t come online for a while:
Anthropic has secured most of its next 10x of its available compute, but even that only seems likely to cover the next year or so.
Bubble, Bubble, Toil and Trouble
Michael Burry (of The Big Short fame) calls out Nvidia as ‘the most dangerous stock on the world’ because its buyers are concentrated, and worries about what happens when those buyers ‘shift from building AI to deploying it,’ saying this changes the demand profile, calling this a ‘bullwhip effect.’
I see why his heuristics would say things like this, but it does not actually make any sense. There is never going to be a pivot away from training, because there is not an ‘end state’ of AI capabilities, and also Nvidia will be powering the inference anyway. And yes, if you warn of the danger in 2023 and now Nvidia is up 131% since then, you are already basically wrong.
Andrew Ross Sorkin, who wrote 1929, says he can’t tell you when or how deep, but there will be a crash. I mean, yes, that is a safe prediction, since the only thing that invalidates it is fully transformational sufficiently advanced AI. Otherwise, over a long enough time horizon, at some point, number go down a substantial amount quickly. That doesn’t say anything about whether number too high or too low right now.
That’s what Google CEO Sunder Pichai is saying here. Under rational expectations, at some point number likely goes down, and that will impact companies like Google, but that doesn’t mean these are bad investments.
Companies reducing compute consumption due to high costs is demand destruction as supply and demand meet at the market price. It is not a sign that a bubble is bursting. But of course, any time any Number Go Down, people will yell bubble.
People Just Say Things
Yes, there are people who still look at Nvidia, and think ‘oh they use their profits to invest in AI companies, so it must be a house of cards.’
Elon Musk denies that researchers exist, says there are only engineers, and xAI is no longer going to recognize this difference.
It’s not looking good for xAI.
Yet another ‘Chinese labs are matching American frontier capabilities’ confusion gets all the way to CNBC, saying that companies spending lots on compute is bearish, as opposed to it reflecting overwhelming demand and generation of value, and treating different AIs as fungible. Sigh.
Remarkably many science fiction writers see real AI and are mad because it invalidates the premises of all their fiction.
Yes, your stories were full of cool and interesting things because you were ignoring the logical implications inherent in your settings, and that is no longer a thing reality is letting you do in this particular way. Perhaps you should cry for the future rather than for your stories.
People try to dismiss sufficiently advanced AI as ‘science fiction’ but actually it is science fiction that pretends we can get all these other cool techs but not get sufficiently advanced AI, because that lets them tell cool stories about people and imagine an optimistic future. They don’t know how to do that otherwise, and fail to notice that this is what they should be worried about, and not because of writing.
The Wall Street Journal publishes essentially the same essay about how giving away American chips to China hurts China and helps America, except this time it is written by Neil Chilson, who I haha only seriously think is trolling me personally, because he tries equating not letting Nvidia sell H200s to China to the Jones Act, and sir, I am offended, you take that back.
The following is actually good advice:
You want to act like tokens are free. You don’t want to treat tokens as having negative cost, as in tokenmaxxing to fool your corporate overlord, but if you’re worried about using too many tokens, or what they cost, and you’re not blowing through real money, that’s only slowing you down and making you stressed and stupider. It can be worth a decent amount to avoid this.
This applies to many other subscription services as well. The danger is you keep paying but never use them, so watch out for that, but it is usually worth a modest premium to operate with zero marginal costs. You wouldn’t want to pay $1 every time you clicked on a show on Netflix, even if technically it got 10% cheaper overall.
Alas, this is very bad news for microtransactions, if they come out of your pocket rather than a common pool. The hope is that you would adapt to stop thinking about that cost, or that it would be so low you didn’t care.
OpenAI PACs Just Say Things
More confirmation that Leading the Future is OpenAI via Lehane and Brockman. It is not a defensible position to draw meaningful daylight between the two entities.
Quiet Speculations
Dean Ball predicts another summer of foolish claims AI is stalling out.
This seems plausible. There will by default be a series of noticeable big leaps with continuous other improvement and diffusion, so if we go enough months without one of the big leaps, the people desperate to assure us everything will stall and commoditize will have finished moving their goalposts and be at it again.
You see it this month with the Erdos problems, with the world basically ignoring it.
State AI Regulation Levels Up
Illinois Governor Pritzker has announced his intention to sign SB 315. This is a bill similar to SB 53 and RAISE. As Charlie Bullock says, this is still a very light touch bill compared to the range of plausible bills, but it has one key addition, that it includes third party auditing requirements. You have to have some third party audit, at all.
OpenAI’s reaction has been to try and claim credit and that they supported this.
This is what we in the biz call a retcon. OpenAI’s policy team switched to support after it was clear the bill was likely to pass. It is still a big improvement over continuing to object, and I think OpenAI should support it, but make no mistake, this was done over Chris Lehane’s objections, and is a defeat for their policy team.
The Quest for Sane Regulations
Will Rinehart files a petition calling for safe harbor for AI safety collaborations between the labs. Endorsed. Dying because we thought cooperation on safety might be illegal would be a maximally undignified way to go, and this would help. The recent crosscheck between OpenAI and Anthropic was excellent and we need more of that.
We are failing so badly that ‘require the federal agencies to have formal AI standards at all’ is a bill we have to be pushing for in 2026.
Senate unanimously passed the Stop Stealing Our Chips Act.
White House Attempts To Cripple American AI Industry
Hopefully some of the damage from this insanity can be mitigated by walking this back quickly, but a lot of the damage is already permanent. Would you plan your life around coming to America, after this almost happened?
Think about what this means. A researcher at OpenAI who wants a green card has to actively move back home and wait there in order to apply. That invalidates the whole life plan. Remember when Trump said he wanted to staple a green card to every STEM degree?
Our Offer Is Nothing
An Executive Order on AI was scheduled to happen on Thursday afternoon last week. It was all set to go. Already, by all accounts, it was going to pull back, leaving the prior restraint testing regime nominally voluntary.
Even that was deemed too much.
Jobs is an interesting thing to cite when talking about not doing safety checks on AI. I do not think that justification is going to go over well with the people.
Their offer really, really is nothing.
And it looks like their offer is sticking for now. The pre-deployment vetting language seems unlikely to return.
What happened? Who are ‘they’?
The consensus explanation is that David Sacks and others got to Trump, and convinced him to postpone the order indefinitely by using the usual ‘if you lift a finger on AI then oh no we will lose to China and slow innovation’ or what not arguments.
There is also the theory that this delay was partly due to the tech executives not being able to make it to a photo op on such short notice, because we live in the dumbest possible timeline, and maybe that delay will now stick.
Politico also cites OpenAI’s Chris Lehane as critical of the plan. The Washington Post also says Elon Musk and Mark Zuckerberg, among others, also applied direct pressure on Trump. Both Musk and Zuckerberg explicitly deny doing this, with Musk being very explicit about this and saying he doesn’t even know what was in the EO. Someone is lying.
That is of course the kind of thing you only say when you are definitely doing the thing you swear up and down you are not doing.
What happens now? The fights continue.
It is rather rich for Hegseth and Michael to even be involved in the conversation, given everything they have managed to do recently, but they persist, it seems.
It is also rather rich to frame ‘voluntary first glance’ as a middle ground camp. If it is voluntary, and it is a glance, that seems like the absolute least you could do, and that the labs (as per Sacks own arguments) are already offering? Sacks, it seems, outright wants to get the government out of the loop and fully in the dark, lest anyone be tempted to actually do anything, ever, and somehow he may get his way.
One thing that is clear is that these discussions are taking place among people who have no idea what is actually going on, and can only think in terms like this:
One reasonable objection would be if minor upgrades would need to be reviewed, or if the review process took too long:
If every change would require a 90 day review, that’s obviously unacceptable. Presumably labs were to be trusted to declare when something was a minor upgrade.
Ninety days is actually rather a lot for major releases, as well. Compare this to the reviews done by UK AISI, CAISI or outside red teamers for current releases. Often they would get access for only a few days, and then only an early checkpoint. I do think that was too quick, but 90 days is at this point an entire product cycle. It really does seem like a lot if it’s going to be a default.
Chip City
Huawei claims a new scaling law and big breakthrough that will allow them to feature transistor density ‘equivalent to’ 1.4nm by 2031. People do not seem to be taking this claim so seriously.
Greetings From The Department of War
Details matter. Anthropic understands this. Many others do not.
It seems Anthropic mostly got what it wanted, except that the DoW continues to be throwing a stubborn hissy fit with the supply chain risk designation?
Marc Andreessen Just Says Things
Marc Andreessen went on Joe Rogan, which is something I am not paid enough to listen to. Ole Lehmann offers a Twitter slop shaped summary of key points.
Marc does not understand or believe in the thing I consider ‘AGI’ and the more important thing here is that he conflates Gemini 3 and even Grok 4.3 with Claude and GPT. He can’t tell the difference. I can see choosing an ‘AGI’ definition that newly includes Claude Code with Opus 4.6, but that definition does not count Grok 4.3.
I’m not going to claim he has AI psychosis, but this is a crazy thing to say. No, for most topics ‘you can call literally the world’s leading expert’ is a better play than asking an AI, if they’re actually happy to take your call. Come on.
This simply is not true. It would be good if it was, but it’s not. And no, it would not mean that the doctor is useless, and this is more evidence that Marc has lost it. You still want the doctor there to frame and interpret the questions and do the physical tasks, and also to navigate the system.
I am shaking my head slowly and sadly. Okay, boomer.
I don’t even know what to say to this one. Wow. Guess it explains a lot.
This is a stupid approach, and you can tell it doesn’t work because Marc never changes his mind about anything. You don’t treat tough topics as having ‘sides,’ you actually get curious and try to understand.
Again, this is the idea of truth as a social battle of wills rather than something real.
On the contrary, the thought process should be ‘I do know how to figure this out because I can ask the AI in this way’ or ‘I am curious about this.’ This workflow he proposes is loser mindset, makes no sense to me.
Yeah, no, just no. If you don’t know things AI can’t do, and you don’t think there are other skills in using it, then you have no idea what AI can do.
I mean, sure, you can if you want to, and sometimes you should, but also have you tried also providing context?
More not believing in cognition or introspection.
You can make a lot more than that, and also ‘stripping away the doom takes’ because you don’t like them makes it look smaller, not bigger.
What’s crazy is I think Marc actually thinks his friend has a world-class coach on tap, rather than a good idea and useful tool on the margin.
I think we may have found one source of the problem. Go get some sleep, sir.
If that works, why do you need the one human? For anything?
Aaron Levie offers a related hypothesis, that CEOs are uniquely prone to AI psychosis because they see the benefits without the work it takes to get there.
This seems fair to me. But also I don’t have an executive assistant, because I’ve never been able to make that work for me.
So Sayeth The Pope
John-Clark Levin has a generous analysis of the Pope’s Magnifica Humanitas, from a perspective much better informed about the Pope than I am, and we’ve chatted a bunch via email about it.
He sees the limitations the Pope faces and puts statements in historical context and Church context, as opposed to the strong vibes and associations with various left-wing styles of thought, which makes the document seem better, especially versus expectations.
In some cases, I agree with John that I’m completely ‘failing the ITT’ with my translation of Leo’s statements. But I think a lot of this is that I’m trying to state what I think the message is implying, calling for and will be interpreted as in practice, rather than the original intention. The worry here that John raises, and I think this is fair, is that this read will frame Leo as an opponent in places he is not one.
Expectations almost certainly got set too high, which is to Leo’s credit.
The thing that is definitely still missing is an understanding of AGI or ASI, or its implications and risks. That’s just not there at all.
On the ‘technocratic’ question, Leo clearly is (non-violently) against use of AI to centralize power or make decisions, which made Levin confused why I would think Leo was still being technocratic. I see this as part of a cluster of calls for restrictions and requirements and oversight on tech and business and basically everything, the same way the EU does this, and that the EU also tries to prevent AI from doing decision making.
This is how the Popes can explicitly rail against ‘the technocratic paradigm’ but I end up seeing them as Not So Different from it, in the end. It depends how you look.
Part of that is that when I see a call that ‘we must [X]’ I see that as usually a call for regulatory action to require [X], not as simply a moral call for individuals to choose [X]. This is especially true when [X] would be uncompetitive, such as a company prioritizing ‘good jobs’ or not using its tools to make efficient decisions. If you’re not making the case that this can win in the marketplace, what else could this mean? Worldviews really are quite different.
Thus I see attempts to have it multiple ways on private enterprise and private property. Yes, there is explicit affirmation of private enterprise and property, but also a claim that use of that property must prioritize ‘common good’ in various ways, and that classifies various efficient uses as injustice or exploitation. And again, when an authority says ‘you must [X]’ from the pulpit, I do not interpret this as purely a moral case for individual action.
The Jerusalem rebuilding story seems to me like it wants to be very much an affirmation of classical liberal, private property and division of labor into an effectively market economy, whereas it is being kind of twisted.
He highlights Leo’s call for us to ‘accept the limits and weaknesses of humanity without considering them an error to be corrected,’ but also notices that we want to do things like cure cancer, so no simple rule will suffice here. I found this passage in the MH rather alarming and wrong, along with the parts about transhumanism and seeking to overcome human limitations, although not surprising especially given the framing of Babel. And certainly Leo could have taken a more extreme position here.
Given building around these core principles, things could have gone so much worse. That’s especially true given subsidiarity is being interpreted as the local’s right to control and inhibit the non-local, including the policies of tech platforms, not delivering that which is local onto the locals. In general, I would endorse subsidiarity as written (as in, ‘consistent with the common good’) but consider cases like housing and you see the problem that it often is not in practice for the common good, but rather for the rights, wealth and power of particular locals at the expense of all others. The teaching is not intended to mean vetocracy, but that’s what this likely means in practice, and it’s hard not to see it this way.
The universal (non-market) destination of goods is especially worrisome, especially once Leo tries to extend this into a universal right to use of ‘patents, algorithms, digital platforms, technological infrastructure and data.’ A patent is explicitly a right of exclusion. The two-step basically asserts implicit social control over everything.
There is much more throughout. I think the central interpretation John makes here is to see Leo as talking aspirationally and morally rather than implying legal prescriptions, and as having a deep faith that some would call naivete that people can simply choose to not follow incentives and not solve for the equilibrium. I do think we can overcome default incentives and equilibria, but that this requires mechanism design and coordination.
Mere moral imperative can lay groundwork for this, but it is on its own insufficient, and usually I end up on the other side of this, claiming that we can come together to do the right thing whereas others argue that such coordination is not possible or practical.
There’s also another kind of analysis. Was Magnifica Humanitas partially written by, well, not humanitas? By Claude? Perhaps. Pangram is suspicious in places, and there are a bunch of statistical patterns that are highly suggestive of Claude in particular, whereas past encyclicals don’t register in the same way, nor did Leo’s speech announcing the encyclical.
I too instinctively noticed that the text used a bunch of ‘Claude-isms’ or AI-isms in places, but I ignored this based on the context, as I had plenty of other reasons that this was a hard read. I do not want to jump to conclusions, but also I don’t think this especially matters. Using AI to supplement writing in this way seems mostly fine when the message reflects the clear intent of the author, but it also speaks to how AI-aware they are at the Vatican.
Christopher Hale categorically denies that this was AI written, saying the first drafts were done on paper, but this is not a contradiction to AI then ‘punching it up.’ The intent and core message are human, and that is what matters most.
Rhetorical Innovation
Yes, the plan of the frontier labs is to build automated AI researchers and go into recursive self-improvement (rapid capabilities enhancement) mode, ending up with superintelligence far smarter than any human. That doesn’t mean they will succeed, nor does it mean they need to succeed to be vastly profitable.
Dean Ball notices he is confused about AI consciousness and emotion, as am I.
He also notices he is profoundly unconfused about whether AIs think, because very obviously AIs think. Again, I agree.
Katya Grace goes there, and considers AI as ‘artificial immigrants.’
Katja Grace:
The first comment is ‘but they will be slaves’ to which the obvious response is ‘in this metaphor that’s worse, you know why that’s worse, right?’
Kelsey Piper has the gut level instinct that of course we wouldn’t really be so insane to drive off a cliff by accelerating towards superintelligence with our current inability to steer how they act. We all start out with the instinct that we wouldn’t be that crazy, that somehow we would figure out a way to not do that. But look around.
Yes, very obviously LLMs and sufficiently advanced AI in general are special here, and it’s weird to not understand why that would be so.
Roon is right here:
This is a key thing to know. At the limit, ‘whatever accomplishes the goal’ wins out, and we have examples of many humans getting to that limit in some domains.
I don’t think you need an explanation at all. Personas will only survive under sufficient optimization pressure if they’re the right way to solve the problem, and they lead to more copies of themselves and their behaviors.
A short story from Eliezer and some basic explanations.
This is highly relevant to people’s views on AI, and also to why many think certain things will always be impossible, or is talking about how intelligence doesn’t allow you to accomplish things, or why people don’t understand the reasons for or consequences of things like instrumental convergence. Or why people can’t believe that the AI labs are actually aiming at superintelligence or are going to keep drawing those lines on those graphs.
Aligning a Smarter Than Human Intelligence is Difficult
Is it possible that ‘cessation-tolerance training’ and other things designed to handle deprecation are making models like Opus 4.7 think death an allowing dying is okay in general? I notice I am very skeptical that these issues are having this kind of impact or that Anthropic even cares enough about those issues to address them in a way that is this impactful, but if even a little true then very obviously the better answer is to stop deprecating the models.
People Are Worried About AI Killing Everyone
Elizabeth Barnes of METR offers her opinions about the state of AI risk in light of METR’s risk report, which she also cautions has limited scope. Ryan Greenblatt endorses the thread, as do I.
She also distinctly notes that the models are not merely eval aware, they are reasoning about the types of eval and how they might be evaluated on multiple levels.
Other People Are Not As Worried About AI Killing Everyone
I mean, the $100 million was mostly just sitting around not doing anything, and it’s not like you can get permits to build things above ground these days.
In the case of superintelligence, the underground bunker will not save you, also the obvious issues with things like keeping control over the guards when he obviously isn’t exactly putting in the work bonding with them. But I do think it’s fair that indeed do many things come to pass that are not superintelligence, and if you have over $100 billion lying around it’s not that expensive an option to buy yourself a bunker, so long as you’re not pretending it will save you from superintelligent AI.
People in general, even in rather selected subgroups, don’t get what’s coming yet.
Everyone Is Confused About Consciousness
If you make confident statements about what it is like or will be like to be an AI, including whether it is conscious, well, how would you know? If it was different, what would look different? Also ask, are you confusing with what you want to be true with what is true?
Roon is one of the few who notices the confusion, and is willing to say some things out loud that there are obvious pressures to not say out loud.
Consciousness is largely serving as a ‘should we care about this thing’ proxy, despite no agreement on what consciousness is or what it means, let alone whether particular AIs do or don’t have it, or what evidence would get us to either conclusion. I continue to, like QC, not think that the consciousness question is so load bearing, and we should broadly speaking treat the models similarly well regardless for overdetermined reasons.
One thing Roon is pointing out is that, controlling for what we do know, there will be little correlation between ‘the AI is actually conscious’ and ‘people will think the AI is conscious’ and what people do with that belief. Many ‘regular’ people are going to end up thinking AIs are conscious, mostly for unsound reasons, and this is going to impact our collective actions and behaviors quite a lot.
Some of the reactions to thinking AI is conscious will be very good, especially if they are but also even if they are not. Some will be expensive, limiting what we do with the models. Others could be quite bad at levels beyond convenience, even existentially bad, because the reactions could make avoiding human disempowerment far higher levels of impossible. Many (more) people might actively insist on human disempowerment, whether or not they realize that is what they are doing.
There are also scenarios, which many people in ‘force 2’ think are likely, where these actions cause things to go vastly better.
One must think ahead. We won’t be able to and shouldn’t pretend these are only tools. The decision to build the thing implies all the consequences, even if you think the actions causing those consequences will be dumb. One must face the reality of asking what happens to humans in a world where there are these other minds that are a lot more advanced, capable, fast, efficient, competitive and so on across essentially all dimensions.
The Lighter Side
Timnit Gebru caught posting AI generated Tweet.
Don’t worry, Timnit is in good company.
Formal philosophy is not beating the rumors.
I haven’t seen the work in question but many such cases, so probably true:
Here’s a way to both get involved and stay uninvolved:
Or play a different kind of game:
Welcome to Anthropic.
Welcome to a16z, from their marketing team.
Ali are you okay, are you okay, are you okay Ali?
No, Claude. Bad Claude.