I suppose it’s better to at least know you need a plan and think to build a bunker, even if you don’t realize that the bunker will do you absolutely no good against the AGI itself, versus not even realizing you need a plan. And the bunker does potentially help against some other threats, especially in a brief early window?
I think Ilya realizes very clearly that the bunker is not against the AGI itself, but only against the turmoil of the "transition period". He seems to be quite explicit in the quoted article ‘We’re Definitely Going to Build a Bunker Before We Release AGI’: The true story behind the chaos at OpenAI (emphasis mine):
“We’re definitely going to build a bunker before we release AGI,” Sutskever replied. Such a powerful technology would surely become an object of intense desire for governments globally. The core scientists working on the technology would need to be protected. “Of course,” he added, “it’s going to be optional whether you want to get into the bunker.”
1
1Why are people talking about AI teachers as if they are good news? The generation taught by AI teachers will probably never be able to get a job. (At least those among them who study teaching.)
It's good news for learning, not necessarily good news for jobs. If you care about creating "teaching" make-work jobs, but don't care whether people know things, then it's bad news.
I don't understand. Only a very small fraction of people are teachers. Education does not exist primarily to make more teachers, it's to help make everything that is productive. Seems great for humans to no longer have to be teachers.
An AI than can replace a teacher in year Y is probably going to replace most white-collar jobs in Y+5. Therefore, when your kids get an AI teacher at school, you should take it as a signal that the job market for humans is going to be over sooner than they finish their education.
That may be a good thing, in case of a nice Singularity. But even then, the education will be irrelevant.
Seems to me that people who look forward to having great AI teachers are imagining a future where we get great AI teachers but everything else remains the same... and then it will be up to the children educated by those great AI teachers to change the world. That future is unlikely to happen.
Yeah, agree with that, though given that I think many good futures route through substantial pauses, or substantial improvements in human coordination technology, mapping out the degree to which AI systems can uplift people before it is capable of disempowering them is a pretty crucial thing to map, so I don't super agree with this equivocation.
Here is another way to defend yourself against bot problems:
Turned out to be fake, BTW. His friend just pranked him.
What a week, huh? America signed a truly gigantic chip sales agreement with UAE and KSA that could be anything from reasonable to civilizational suicide depending on security arrangements and implementation details, Google announced all the things, OpenAI dropped Codex and also bought Jony Ive’s device company for $6.5 billion, Vance talked about reading AI 2027 (surprise, in a good way!) and all that other stuff.
Lemon, it’s Thursday, you’ve got movie tickets for Mission Impossible: Final Reckoning (19th and Broadway AMC, 3pm), an evening concert tonight from Light Sweet Crude and there’s a livestream from Anthropic coming up at 12:30pm eastern, the non-AI links are piling up and LessOnline is coming in a few weeks. Can’t go backwards and there’s no time to spin anything else out of the weekly. Got to go forward to go back. Better press on.
So for the moment, here we go.
Table of Contents
Earlier this week: Google I/O Day was the ultimate ‘huh, upgrades’ section. OpenAI brought us their Codex of Ultimate Vibing (and then Google offered their version called Jules). xAI had some strong opinions strongly shared in Regarding South Africa. And America Made a very important AI Chip Diffusion Deal with UAE and KSA, where the details we don’t yet know could make it anything from civilizational suicide to a defensible agreement, once you push back the terrible arguments made in its defense.
Language Models Offer Mundane Utility
AI scientist announces potential major discovery, a promising treatment for dry AMD, a major cause of blindness. Paper is here.
Nikhil Krishnan sees health care costs going up near term due to AI for three reasons.
This seems right in the near term. The entire health care system is bonkers and bans real competition. This is the result. In the medium term, it should radically improve health care productivity and outcomes, and then we can collectively decide how much to spend on it all. In the long term, we will see radical improvements, or we won’t need any health care.
In a related story, ChatGPT helps students feign ADHD. Well, not really. The actual story is ‘a 2000 word document created via ChatGPT, in a way that ordinary prompting would not easily duplicate, helps students feign ADHD.’ So mostly this is saying that a good guide helps you fake ADHD, and that with a lot of effort ChatGPT can produce one. Okie dokie.
Let’s check in on AlphaEvolve, a name that definitely shouldn’t worry anyone, with its results that also definitely shouldn’t worry anyone.
AI improves European weather forecasts 20% on key indicators. Progress on whether forecasters is also impressive, but harder to measure.
AI helping executives handle their inboxes and otherwise sift through overwhelming amounts of incoming information. My read is the tools are just now getting good enough that power users drowning in incoming communications turn a profit, but not quite good enough for regular people. Yet.
Language Models Don’t Offer Mundane Utility
As usual, that’s if you dismiss them out of hand and don’t use them, such as Judah Diament saying this is ‘not a breakthrough’ because ‘there have been such tools since the late 1980s.’ What’s the difference between vibe coding and Microsoft Visual Basic, really, when you dig down?
Curio AI stuffed toys, which seem a lot like a stuffed animal with an internet connection to a (probably small and lame) AI model tuned to talk to kids, that has a strict time limit if you don’t pay for a subscription beyond 60 days?
MIT economics department ‘conducted an internal, confidential review’ of this paper and concluded it ‘should be withdrawn from public discourse.’ It then clarifies this was due to misconduct, and that the author is no longer at MIT, and that this was due to ‘concerns about the validity of the research.’
Here is abstract of the paper that we should now treat as not real, as a reminder to undo the update you made when you saw it:
That was a very interesting claim, but we have no evidence that it is true. Or false.
I was going to call MIT’s statement ‘beating around the bush’ the way this WSJ headline does saying MIT ‘can no longer stand behind’ the paper, but no, to MIT’s credit they very clearly are doing everything their lawyers will allow them to do, the following combined with the student leaving MIT is very clear:
It seems so crazy to me that ‘student privacy’ should bind us this way in this spot, but here we are. Either way, we got the message. Which is, in English:
A (not new) theory of why Lee Sedol’s move 78 caused AlphaGo to start misfiring, where having a lot of similar options AlphaGo couldn’t differentiate between caused it to have to divide its attention into exponentially many different lines of play. My understanding is it was also objectively very strong and a very unlikely move to have been made, which presumably also mattered? I am not good enough at Go to usefully analyze the board.
Paper finds LLMs produce ‘five times less accurate’ summaries of scientific research than humans, warning of ‘overgeneralization’ and omission of details that limit scope. All right, sure, and that’s why you’re going to provide me with human summaries I can use instead, right, Anakin? Alternatively, you can do what I do and ask follow-up questions to check on all that.
DeepSeek powers a rush of Chinese fortune telling apps, in section IV of the type of article, here on the rise of Chinese superstitious and despairing behavior, that could be charting something important but could easily be mostly hand picked examples. Except for the rise in scratch-off lottery tickets, which is a hugely bearish indicator. I also note that it describes DeepSeek as ‘briefly worrying American tech companies,’ which is accurate, except that the politicians don’t realize we’ve stopped worrying.
Huh, Upgrades
Claude’s Research now available on mobile, weird that it wasn’t before.
Some changes were made to the Claude 3.7 system prompt.
xAI’s API now can search Twitter and the internet, like everyone else.
Codex of Ultimate Vibing
Some more takes on Codex:
Sunless highlights that in many ways the most valuable time for something like Codex is right after you get access. You can use it to suddenly do all the things you had on your stack that it can easily do, almost for free, that you couldn’t do easily before. Instant profit. It may never feel that good again.
I strongly agree with Diamond’s second and third points here. If you close the IDE afterwards you’re essentially saying that you should assume it’s all going to work, so it’s fine to have to redo a bunch of work if something goes wrong. That’s a terrible assumption. And it’s super hard to test without internet access.
On Your Marks
How big a deal is AlphaEvolve? Simeon thinks it is a pretty big deal, and most other responses here agree. As a proof of concept, it seems very important to me, even if the model itself doesn’t do anything of importance yet.
Choose Your Fighter
How OpenAI suggests you choose your model.
In practice, my answer is ‘o3 for everything other than generating images, unless you’re hitting your request limits, anything where o3 is the wrong choice you should be using Claude or Gemini.’
Seriously, I have a harder and harder time believing anyone actually uses Grok, the ultimate two-handed language model.
Deepfaketown and Botpocalypse Soon
This is indeed how it feels these days.
Instantaneously we can see that this is ‘wrong’ and therefore AI, then over the course of a minute you can extract particular reasons why. It’s like one of those old newspaper exercises, ‘spot all the differences in this picture.’
I find the obviously fake art here does make me less inclined to eat here. I don’t want you to spend a ton of time on marketing, but this is exactly the wrong way and amount to care, like you wanted to care a lot but didn’t have the budget and you aren’t authentic or detail oriented. Stay away. The vibe doesn’t jive with caring deeply about the quality of one’s pizza.
Since IGN already says what I’d say about this, I turn over the floor:
Actually, after watching the video, it’s going way better than expected. Love it.
Here is another way to defend yourself against bot problems:
Is it morally wrong to create and use fully private AI porn of someone who didn’t consent? Women overwhelmingly (~10:1) said yes, men said yes by about 2.5:1.
I don’t think that’s it. I think we are considering this immoral partly because we think (rightly or wrongly) that porn and sex and even thinking about other people sexually (even with permission and especially without it) is gross and immoral in general even if we don’t have a way to ban any of it. And often we try anyway.
Even more central, I think, is that we don’t trust anything private to stay truly private, the tech is the same for private versus public image (or in the future video or even VR!) generation, we have a concept of ownership over ‘name and likeness,’ and we don’t want to give people the ‘it was only private’ excuse.
Not AI but worth noting: Ben Jacobs warns about a scam where someone gets control of a contact’s (real) Telegram, invites you to a meeting, then redirects you to a fake zoom address which asks you to update zoom with a malicious update. I recommend solving this problem by not being on Telegram, but to each their own.
Ideally we’d also be warning the scammers.
Copyright Confrontation
The creatives continue to be restless. Morale has not improved.
That’s not what ‘criminal offense’ means, but point taken.
Regarding South Africa
Zeynep Tufekci writes up what happened to Grok in the New York Times, including providing a plausible triggering event to explain why the change might have been made on that particular day, and ties it to GPT-4o being an absurd sycophant as a general warning about what labs might choose to do with their bots. This, it seems, is what causes some to worry about the ‘safety’ of bots. Okay then.
Cheaters Gonna Cheat Cheat Cheat Cheat Cheat
And those not cheating will use AI too, if only to pass the AI filters? Oh boy. I mean, entirely unsurprising, but oh boy.
The tragedy of all this is that when they do catch someone using AI, they typically get away with it, but still everyone has to face this police state of running everything through the checkers.
It also highlights that your AI checker has to be able to defeat a student who has access to an AI checker. Right now the system is mostly not automated, but there’s nothing stopping one from creating a one-button agent that takes an essay – whether it was an AI or a human that wrote the original – feeding it into the public AI detector, and then iterating as needed until the essay passes. It would then be insane not to use that, and ‘who gets detected using AI’ by default becomes only those who don’t know to do that.
The only way to get around this is to have the AI checker available to teachers be superior to the one used by students. It’s like cybersecurity and other questions of ‘offense-defense balance.’ And it is another illustration of why in many cases you get rather nasty results if you simply open up the best functionality to whoever wants it. I don’t see a way to get to a future where this particular ‘offense-defense balance’ can properly favor the AI detectors actually catching cheaters.
Unless? Perhaps we are asking the wrong question. Rather than ask ‘did an AI write this?’ you could ask ‘did this particular student write this?’ That’s a better question. If you can require the student to generate writing samples in person that you know are theirs, you can then do a comparison analysis.
Tyler Cowen bites all the bullets, and says outright ‘everyone’s cheating, that’s good news.’ His view is essentially that the work the AI can do for you won’t be valuable in the future, so it’s good to stop forcing kids to do that work. Yes, right now this breaks the ‘educational system’ until it can adjust, but that too is good, because it was already broken, it has to change and it will not go quietly.
As is typically true with Tyler, he gets some things that AI will change, but then assumes the process will stop, and the rest of life will somehow continue as per normal, only without the need for the skills AI currently is able to replace?
It is hard for me to picture the future world Tyler must be imagining, with any expectation it would be stable.
If you are assigning two-month engineering problems to students, perhaps check if Gemini 2.5 can spit out the answer. Yes, this absolutely is the ‘death of this type of coursework.’ That’s probably a good thing.
Something tells me that ‘ChatGPT’ here probably wasn’t o3?
They Took Our Jobs
In a new study from Jung Ho Choi and Chloe Xie, AI allowed accountants to redirect 8.5% of their time away from data entry towards other higher value tasks and resulted in a 55% increase in weekly client support.
Notice what happens when we decompose work into a fixed cost in required background tasks like data entry, and then this enables productive tasks. If a large percentage of time was previously data entry, even a small speedup in that can result in much more overall productivity.
This is more generally true than people might think. In most jobs and lives, there are large fixed maintenance costs, which shrinks the time available for ‘real work.’ Who among us spends 40 hours on ‘real work’? If you speed up the marginal real work by X% while holding all fixed costs fixed, you get X% productivity growth. If you speed up the fixed costs too, you can get a lot more than X% total growth.
This also suggests that the productivity gains of accountants are being allocated to increased client support, rather than into each accountant serving more clients. Presumably in the long term more will be allocated towards reducing costs.
The other big finding is that AI and accountants for now remain complements. You need an expert to catch and correct errors, and guide the AI. Over time, that will shift into the AI both speeding things up more and not needing the accountant.
At Marginal Revolution, commenters find the claims plausible. Accounting seems like a clear example of a place where AI should allow for large gains.
Tyler Cowen also links us to Dominic Coey who reminds us that Baumol’s Cost Disease is fully consistent with transformative economic growth, and to beware arguments from cost disease. Indeed. If AI gives us radically higher productivity in some areas but not others, we will be vastly richer and better off. Indeed in some ways this is ideal because it lets us still have ‘jobs.’
It is a question of when, not if. It’s always a skill issue, for some value of skill.
A hypothesis that many of the often successful ‘Substack house style’ essays going around Substack are actually written by AI. I think Will Storr here has stumbled on a real thing, but that for now it is a small corner of Substack.
Robert Scoble provides us another example of what we might call ‘human essentialism.’ He recognizes and expects we will likely solve robotics within 10 years and they will be everywhere, we will have ‘dozens of virtual beings in our lives,’ expects us to use a Star Trek style interface with computers without even having applications. But he still thinks human input will be vital, that it will be AIs and humans ‘working together’ and that we will be ‘more productive’ as if the humans are still driving productivity.
I don’t see these two halves of his vision as compatible, even if we do walk this ‘middle path.’ If we have robots everywhere and don’t need 2D screens or keyboards or apps, what are these ‘new things to do’ that the AI can’t do itself? Even if we generously assume humans find a way to retain control over all this and all existential-style worries and instability fall away, most humans will have nothing useful to contribute to such a world except things that rely on their human essentialism – things were the AI could do it, but the AI doing it would rob it of its meaning, and we value that meaning enough to want the thing.
They took our jobs and hired the wrong person?
The models all also favored whoever was listed first and candidates with pronouns in bio. David interprets this as LLMs ‘not acting rationally,’ instead articulating false reasons that don’t stand up to scrutiny.
And yes, all of that is exactly like real humans. The AI is correctly learning to do some combination of mimic observed behavior and read the signs on who should be hired. But the AIs don’t want to offer explicit justifications of that any more than I do right now, other than to note that whoever you list first is sometimes who you secretly like better and AI can take a hint because it has truesight, and it would be legally problematic to do so in some case, so they come up with something else.
Tyler Cowen calls this ‘politically correct LLMs’ and asks:
This is inherent in the data set, as you can see from it appearing in every model, and of course no one is trying to get the AIs to take the first listed candidate more often. If you don’t like this (or if you do like it!) do not blame it on alignment work. It is those who want to avoid these effects who want to put an intentional thumb on the scale, whether or not we find that desirable. There is work to do.
Scott Lincicome asks, what if AI means more jobs, not fewer? Similar to the recent comments by JD Vance, it is remarkable how much such arguments treat the prior of ‘previous technologies created jobs’ or ‘AI so far hasn’t actively caused massive unemployment’ as such a knock-down arguments that anyone doubting them is being silly.
Perhaps a lot of what is going on is there are people making the strawman-style argument that AI will indeed cause mass unemployment Real Soon Now, and posts like this are mainly arguing against that strawman-style position. In which case, all right, fair enough. Yet it’s curious how such advocates consistently try to bite the biggest bullets along the way, Vance does it for truck drivers and here Scott chooses radiologists, where reports of their unemployment have so far been premature.
While AI is offering ‘ordinary productivity improvements’ and automating away some limited number of jobs or tasks, yes, this intuition likely holds, and we won’t have an AI-fueled unemployment problem. But as I keep saying, the problem comes when the AI also does the jobs and tasks you would transfer into.
The Art of the Jailbreak
Here’s the Gemini Diffusion system prompt.
Get Involved
Anthropic hosting a social in NYC in mid-June for quants considering switch careers, submissions due June 9th.
Job as an AI grantmaker at Schmidt Sciences.
Georgetown offering research funding from small size up to $1 million for investigation of dangers from internal deployment of AI systems. Internal deployment seems like a highly neglected threat model. Expressions of interest (~1k words) due June 30, proposal by September 15. Good opportunity, but we need faster grants.
A draft of a proposed guide for whistleblowers (nominally from AI labs, but the tactics look like they’d apply regardless of where you work), especially those who want to leave the USA and leak classified information. If the situation does pass the (very very high!) bar for justifying this, you need to do it right.
In Other AI News
Google One now has 150 million subscribers, a 50% gain since February 2024. It is unclear the extent to which the Gemini part of the package is driving subscriptions.
The Waluigi Effect comes to Wikipedia, also it has a Wikipedia page.
Financial Times reports that leading models have a bias towards their own creator labs and against other labs, but Rob Wiblin observes that this bias does not seems so large:
This seems about as good as one could reasonably expect? But yes there are important differences. Notice that Altman’s description here has his weakness as ‘the growing perception that’ he is up to no good, whereas Sonnet and several others suggest it is that Altman might actually be up to no good.
Vanity Fair: Microsoft CEO Satya Nadella Explains How He’s Making Himself Obsolete With AI. If anything it seems like he’s taking it too far too fast.
Remember that time Ilya Sutskever said OpenAI were ‘definitely going to build a bunker before we release AGI’?
I suppose it’s better to at least know you need a plan and think to build a bunker, even if you don’t realize that the bunker will do you absolutely no good against the AGI itself, versus not even realizing you need a plan. And the bunker does potentially help against some other threats, especially in a brief early window?
The rest of the post is about various OpenAI troubles that led to and resulted in and from The Battle of the Board, and did not contain any important new information.
Reports of a widening data gap between open and closed models, seems plausible:
Mark Gurman and Drake Bennett analyze how Apple’s AI efforts went so wrong, in sharp contrast to Google’s array of products on I/O day. ‘This is taking a bit longer than expected’ is no longer going to cover it. Yes, Apple has some buffer of time, but I see that buffer running low. They present this as a cultural mismatch failure, where Apple was unwilling to invest in AI properly until it knew what the product was, at which point it was super fall behind, combined with a failure of leadership and their focus on consumer privacy. They’re only now talking about turning Siri ‘into a ChatGPT competitor.’
Much Ado About Malaysia
It isn’t actually meaningful news, but it is made to sound like it is, so here we are: Malaysia launches what it calls the region’s ‘first sovereign full-stack AI infrastructure,’ storing and managing all data and everything else locally in Malaysia.
They will use locally run models, including from DeepSeek since that is correctly the go-to open model because OpenAI’s hasn’t released yet, Meta is terrible and Google has failed marketing forever. But of course they could easily swap that if a better one becomes available, and the point of an open model is that China has zero control over what happens in Malaysia.
Malaysia is exactly the one country I singled out, outside of the Middle East, as an obvious place not to put meaningful quantities of our most advanced AI chips. They don’t need them, they’re not an important market, they’re not important diplomatically or strategically, they’re clearly in China’s sphere of influence and more allied to China than to America, and they have a history of leaking chips to China.
And somehow it’s the place that Sacks and various companies are touting as a place to put advanced AI chips. Why do you think that is? What do you think those chips are for? Why are we suddenly treating selling Malaysia those chips as a ‘beat China’ proposal?
They are trying to play us, meme style, for absolute fools.
And yet, here we are, with Sacks trying to undermine his own administration in order to keep the chips flowing to China’s sphere of influence. I wonder why.
It’s one thing to argue we need a strategic deal with UAE and KSA. I am deeply skeptical, we’ll need a hell of a set of security procedures and guarantees, but one can make a case that we can get that security, and that they bring a lot to the table, and that they might actually be and become our friends.
But Malaysia? Who are we even kidding? They have played us for absolute fools.
It almost feels intentional, like those who for some unknown reason care primarily about Nvidia’s market share and profit margins choosing the worst possible example to prove to us exactly what they actually care about. And by ‘they’ I mean David Sacks and I also mean Nvidia and Oracle.
But also notice that this is a very small operation. One might even say it is so small as to be entirely symbolic.
The original announced intent was to use only 3,000 Huawei chips to power this, the first exported such chips. You know what it costs to get chips that could fill in for 3,000 Ascend 910Cs?
About 14 million dollars. That’s right. About 1% of what Malaysia buys in chips from Taiwan and America each month right now, as I’ll discuss later. It’s not like they couldn’t have done that under Biden. They did do that under Biden. They did it every month. What are we even talking about?
I presume that, since this means the Malaysian government is announcing to the world that it is directly violating our export controls, combined with previous smuggling of chips out of Malaysia having been allowed, we’re going to cut them off entirely from our own chips? Anakin?
It’s weird, when you combine all that, to see this used as an argument against the diffusion rules, in general, and that the administration is telling us that this is some sort of important scary development? These words ‘American AI stack’ are like some sort of magical invocation, completely scope insensitive, completely not a thing in physical terms, being used as justification to give away our technology to perhaps the #1 most obvious place that would send those chips directly to the PCR and has no other strategic value I can think of?
This would be the literal first time that any country on Earth other than China was deploying Huawei chips at all.
And it wasn’t even a new announcement!
One might even say that the purpose of this announcement was to give ammunition to people like Sacks to tout the need to sell billions in chips where they can be diverted. The Chinese are behind, but they are subtle, they think ahead and they are not dumb.
For all this supposed panic over the competition, the competition we fear so much that Nvidia says is right on our heels has deployed literally zero chips, and doesn’t obviously have a non-zero number of chips available to deploy.
So we need to rush to give our chips to these obviously China-aligned markets to ‘get entrenched’ in those markets, even though that doesn’t actually make any sense whatsoever because nothing is entrenched or locked in, because in the future China will make chips and then sell them?
And indeed, Malaysia has recently gone on a suspiciously large binge buying American AI chips, with over a billion in purchases each in March and April? As in, even with these chips our ‘market share’ in Malaysia would remain (checks notes) 99%.
I told someone in the administration it sounded like they were just feeding American AI chips to China and then I started crying?
I’ve heard of crazy ‘missile gap’ arguments, but this has to be some sort of record.
But wait, there’s more. Even this deal doesn’t seem to involve Huawei after all?
Will we later see a rash of these ‘sovereign AI’ platforms? For some narrow purposes that involve sufficiently sensitive data and lack of trust in America I presume that we will, although the overall compute needs of such projects will likely not be so large, nor will they mostly require models at the frontier.
And there’s no reason to think that we couldn’t supply such projects with chips in the places it would make any sense to do, without going up against the Biden diffusion rules. There’s no issue here.
Update your assessment of everyone’s credibility and motives accordingly.
Show Me the Money
LMArena raises $100 million at a $600 million valuation, sorry what, yes of course a16z led the funding round, or $20 per vote cast on their website, and also I think we’re done here? As in, if this wasn’t a bought and paid for propaganda platform before, it sure as hell is about to become one. The price makes absolutely no sense any other way.
OpenAI buys AI Device Startup from Jony Ive for $6.5 billion, calls Ive ‘the deepest thinker Altman’s ever met.’ Jony Ive says of his current prototype, ‘this is the best work our team has ever done,’ this from a person who did the iPhone and MacBook Pro. So that’s a very bold claim. The plan is for OpenAI to develop a family of AI-powered devices to debut in 2026, shipping over 100 million devices. They made a nine minute announcement video. David Lee calls it a ‘long-shot bet to kill the iPhone.’
Quiet Speculations
Great expectations, coming soon, better to update later than not at all.
What do they plan to do about this, to prepare for this future? Um… have a flexible budget, whatever that means? Make some investments, maybe? I wonder what is on television.
Here are some better-calibrated expectations, as METR preliminarily extends its chart of how fast various AI capabilities are improving.
Will future algorithmic progress in an intelligence explosion be bottlenecked by compute? Epoch AI says yes, Ryan Greenblatt says no. In some sense everything is bottlenecked by compute in a true intelligence explosion, since the intelligences work on compute, but that’s not the question here. The question is, will future AIs be able to test and refine algorithmic improvements without gigantic test compute budgets? Epoch says no because Transformers, MoE and MQA are all compute-dependent innovations. But Ryan fires back that all three were first tested and verified at small scale. My inclination is strongly to side with Ryan here. I think that (relatively) small scale experiments designed by a superintelligence should definitely be sufficient to choose among promising algorithmic candidates. After I wrote that, I checked and o3 also sided mostly with Ryan.
New paper in Science claims decentralized populations of LLM agents develop spontaneous universally adopted social conventions. Given sufficient context and memory, and enough ‘social’ interactions, this seems so obviously true I won’t bother explaining why. But the study itself is very clearly garbage, if you read the experimental setup. All it is actually saying is if you explicitly play iterated pairwise coordination games (as in, we get symmetrically rewarded if our outputs match), agents will coordinate around some answer. I mean, yes, no shit, Sherlock.
Popular Mechanics writes up that Dario Amodei and other tech CEOs are predicting AI will allow humans to soon (as in, perhaps by 2030!) double the human lifespan or achieve ‘escape velocity,’ meaning a lifespan that increases faster than one year per year, allowing us to survive indefinitely.
I’d be happy to bet against it too if the deadline is 2030. This is a parlay, a bet on superintelligence and fully transformational AI showing up before 2030, combined with humanity surviving that, and that such life extension is physically feasible and we are willing to implement and invest in the necessary changes, all of which would have to happen very quickly. That’s a lot of ways for this not to happen.
However, most people are very much sleeping on the possibility of getting to escape velocity within our lifetimes, as in by 2040 or 2050 rather than 2030, which potentially could happen even without transformational AI, we should fund anti-aging research. These are physical problems with physical solutions. I am confident that with transformational AI solutions could be found if we made it a priority. Of course, we would also have to survive creating transformational AI, and retain control sufficiently to make this happen.
Nikita Bier predicts that AI’s ability to understand text will allow much more rapid onboarding of customization necessary for text-based social feeds like Reddit or Twitter. Right now, such experiences are wonderful with strong investment and attention to detail, but without this they suck and most people won’t make the effort. This seems roughly right to me, but also it seems like we could already be doing a much better job of this, and also based on my brief exposure the onboarding to TikTok is actually pretty rough.
What level of AI intelligence or volume is required before we see big AI changes, and how much inference will we need to make that happen?
Eliezer’s point here confused some people, but I believe it is that if AI is about as intelligent as the average human and you are trying to slot it in as if it was a human, and you have only so many such AIs to work with due to limits to algorithmic improvements, say 114 million in 2028, then 25% growth per year, then you would only see big improvements to the extent the AI was able to do things those humans couldn’t. And Patel is saying that depends more on other factors than intelligence. I think that’s a reasonable position to have on the margins being discussed here, where AI intelligence is firmly in the (rather narrow) normal human range.
However, I also think this is a clearly large underestimate of the de facto number of AIs we would have available in this spot. An AI only uses compute during active inference or training. A human uses their brain continuously, but most of the time the human isn’t using it for much, or we are context shifting in a way that is expensive for humans but not for AIs, or we are using it for a mundane task where the ‘required intelligence’ for the task detail being done is low and you could have ‘outsourced that subtask to a much dumber model.’ And while AI is less sample-efficient at learning than we are, it transfers learning for free and we very, very much don’t. This all seems like at least a 2 OOM (order of magnitude) effective improvement.
I also find it highly unlikely that the world could be running on compute in 2028, we hit the TSMC wafer limit, and using even those non-superintelligent AIs and the incentives to scale them no one figures out a way to make more wafers or otherwise scale inference compute faster.
Autonomous Dancing Robots
The humanoid robots keep rapidly getting better, at the link watch one dance.
Plenty of people were willing to disprove this claim via counterexample.
I would find it very surprising if, were this to become highly affordable and capable of doing household chores well, it didn’t become the default to have one. And I think Robert is super on point, having robots that can do arbitrary ‘normal’ physical tasks will be a complete lifestyle game changer, even if they are zero percent ‘creative’ in any way and have to be given specific instructions.
Frankly I’d be tempted to buy one if it even if literally all it could do was dance.
The Quest for Sane Regulations
A general reminder that Congress is attempting to withdraw even existing subsidies to building more electrical power capacity. If we are hard enough up for power to even consider putting giant data centers in the UAE, the least we could do is not this?
Alasdair Phillips-Robins and Sam Winter-Levy write a guide to knowing whether the AI Chips deal was actually good. As I said last week, the devil is in the details. Everything they mention here falls under ‘the least you could do,’ I think we can and must do a lot better than this before I’d be fine with a deal of this size. What I especially appreciate is that giving UAE/KSA the chips should be viewed as a cost, that we pay in order to extract other concessions, even if they aren’t logically linked. Freezing China out of the tech stack is part of the deal, not a technical consequence of using our chips, the same way that you could run Gemma or Llama on Huawei chips.
It’s insane I have to keep quoting people saying this, but here we are:
David Sacks attempts to blame our failure to Build, Baby, Build on the Biden Administration, in a post with improved concreteness. I agree that Biden could have been much better at turning intention into results, but what matters is what we do now. When Sacks says the Trump administration is ‘alleviating the bottlenecks’ what are we actually doing here to advance permitting reform and energy access?
Everyone seems to agree on this goal, across the aisle, so presumably we have wide leeway to not only issue executive orders and exemptions, but to actually pass laws. This seems like a top priority.
The other two paragraphs are repetition of previous arguments, that lead to questions we need better answers to. A central example is whether American buildout of data centers is actually funding constrained. If it is, we should ask why but welcome help with financing. If it isn’t, we shouldn’t be excited to have UAE build American data centers, since they would have been built anyway.
And again with ‘Huawei+DeepSeek,’ what exactly are you ‘selling’ with DeepSeek? And exactly what chips is China shipping with Huawei, and are they indeed taking the place of potential data centers in Beijing and Shanghai, given their supply of physical chips is a limiting factor? And if China can build [X] data centers anywhere, should it concern us if they do it in the UAE over the PRC? Why does ‘the standard’ here matter when any chip can run any model or task, you can combine any set of chips, and model switching costs are low?
In his interview with Ross Douthat, VP Vance emphasized energy policy as the most important industrial policy for America, and the need to eliminate regulatory barriers. I agree, but until things actually change, that is cheap talk. Right now I see a budget that is going to make things even worse, and no signs of meaningfully easing permitting or other regulatory barriers, or that this is a real priority of the administration. He says there is ‘a lot of regulatory relief’ in the budget but I do not see the signs of that.
If we can propose, with a straight face, an outright moratorium on enforcing any and all state bills about AI, how about a similar moratorium on enforcing any and all state laws restricting the supply of electrical power? You want to go? Let’s f***ing go.
The Mask Comes Off
We now have access to a letter that OpenAI sent to California Attorney General Rob Bonta.
What did we learn that we didn’t previously know, about OpenAI’s attempt to convert itself into a PBC and sideline the nonprofit without due compensation?
First of all, Garrison Lovely confirms the view Rob Wilbin and Tyler Whitmer have, going in the same direction I did in my initial reaction, but farther and with more confidence that OpenAI was indeed up to no good.
Here is his view on the financing situation:
There is no contradiction here. OpenAI’s valuation in that round was absurdly low if you had been marketing OpenAI as a normal corporation. A substantial price was paid. They did fill the round to their satisfaction anyway with room to spare, at this somewhat lower price and with a potential refund offer. This was nominally conditional on a conversion, but that’s a put that is way out of the money. OpenAI’s valuation has almost doubled since then. What is SoftBank going to do, ask for a refund? Of course nothing has changed.
The most important questions about the restructuring are: What will the nonprofit actually have the rights to do? And what obligations to the nonprofit mission will the company and its board have?
A ‘substantial stake’ is going to no doubt be a large downgrade in their expected share of future profits, the question is how glaring a theft that will be.
The bigger concern is control. The nonprofit board will go from full direct control to the ability to fire PBC directors. But the power to fire the people who decide X is very different from directly deciding X, especially in a rapidly evolving scenario, and when the Xs have an obligation to balance your needs with the maximization of profits. This is a loss of most of the effective power of the nonprofit.
The way I put this before was: The new arrangement helps Sam Altman and OpenAI do the right thing if they want to do the right thing. If they want to do the wrong thing, this won’t stop them.
As Tyler Whitmer discusses on 80,000 Hours, it is legally permitted to write into the PBC’s founding documents that the new company will prioritize the nonprofit mission. It sounds like they do not intend to do that.
OpenAI has, shall we say, not been consistently candid here. The letter takes a very hard stance against all critics while OpenAI took a public attitude of claiming cooperation and constructive dialogue. It attempts to rewrite the history of Altman’s firing and rehiring (I won’t rehash those details here). It claims ‘the nonprofit board is stronger than ever’ (lol, lmao even). It claims that when the letter ‘Not For Private Gain’ said OpenAI planned to eliminate nonprofit control that this was false, while their own letter elsewhere admitted this was indeed exactly OpenAI’s plan, and then when they announced their change in plans characterized the change as letting the board remain in control, thus admitting this again, while again falsely claiming the board would retain its control.
Garrison also claims that OpenAI is fighting dirty against its critics beyond the contents of the letter, such as implying they are working with with Elon Musk when OpenAI had no reason to think this was not the case, and indeed I am confident it is not true.
The Week in Audio
Yoshua Bengio TED talk on his personal experience fighting AI existential risk.
Rowan Cheung interviews Microsoft CEO Satya Nadella, largely about agents.
Demis Hassabis talks definitions of AGI. If the objection really is ‘a hole in the system’ and a lack of consistency in doing tasks, then who among us is a general intelligence?
As referenced in the previous section, Rob Wiblin interviews litigator Tyler Whitmer of the Not For Private Gain coalition. Tyler explains that by default OpenAI’s announcement that ‘the nonprofit will retain control’ means very little, ‘the nonprofit can fire the board’ is a huge downgrade from their current direct control, this would abrogate all sorts of agreements. In a truly dangerous scenario, having to go through courts or otherwise act retroactively comes too late. And we can’t even be assured the ‘retaining control’ means even this minimal level of control.
This is all entirely unsurprising. We cannot trust OpenAI on any of this.
The flip side of the devil being in the details is that, with the right details, we can fight to get better details, and with great details, in particular writing the non-profit mission in as a fiduciary duty of the board of the new PBC, we can potentially do well. It is our job to get the Attorney Generals to hold OpenAI to account and ensure the new arrangement have teeth.
Ultimately, given what has already happened, the best case likely continues to mostly be ‘Sam Altman has effective permission to do the right thing if he chooses to do it, rather than being legally obligated to do the wrong thing.’ It’s not going to be easy to do better than that. But we can seek to at least do that well.
Kevin Roose reflects on Sydney, and how we should notice how epic are the fails even from companies like Microsoft.
Will OpenAI outcompete startups? Garry Tan, the head of YC, says no. You have to actually build a business that uses the API well, if you do there’s plenty of space in the market. For now I agree. I would be worried that this is true right up until it isn’t.
Write That Essay
You’d be surprised who might read it.
In the case of Situational Awareness, it would include Ivanka Trump.
In the case of AI 2027, it would be Vice President JD Vance, among the other things he said in a recent interview with Ross Douthat that was mostly about immigration.
It is true that I probably should be trying harder to write things in this reference class. I am definitely writing some things with a particular set of people, or in some cases one particular person, in mind. But the true ‘essay meta’ is another level above that.
Vance on AI
What else did Vance say about AI in that interview?
First, in response to being asked, he talks about jobs, and wow, where have I heard these exact lines before about how technology always creates jobs and the naysayers are always wrong?
I consider that a zombie argument in the context of AI, and I agree (once again) that up to a point when AI takes over some jobs we will move people to other jobs, the same way bank tellers transitioned to other tasks, and all that. But once again, the whole problem is that when the AI also takes the new job you want to shift into, when a critical mass of jobs get taken over, and when many or most people can’t meaningfully contribute labor or generate much economic value, this stops working.
Then we get into territory that’s a lot less realistic.
I’m sorry, what? You think we’re going to have self-driving trucks, and we’re not going to employ less truck drivers?
I mean, we could in theory do this via regulation, by requiring there be a driver in the car at all times. And of course those truck drivers could go do other jobs. But otherwise, seriously, who are you kidding here? Is this a joke?
I actually agree with Vance that economic concerns are highly secondary here, if nothing else we can do redistribution or in a pinch create non-productive jobs.
So let’s move on to Vance talking about what actually bothers him. He focuses first on social problems, the worry of AI as placebo dating app on steroids.
It seems weird to think that the three million truck drivers will still be driving trucks after those trucks can drive themselves, but that’s a distinct issue from what Vance discusses here. I do think Vance is pointing to real issues here, with no easy answers, and it’s interesting to see how he thinks about this. In the first half of the interview, he didn’t read to me like a person expressing his actual opinions, but here he does.
Then, of course, there’s the actual big questions.
Those are indeed good things to worry about. And then it gets real, and Vance seems to be actually thinking somewhat reasonably about the most important questions, although he’s still got a way to go?
Fair enough. Asking for a unilateral pause is a rough ask if you take the stakes sufficiently seriously, and think things are close enough that if you pause you would potentially lose. But perhaps we can get into a sufficiently strong position, as we do in AI 2027. Or we can get China to follow along, which Vance seems open to. I’ll take ‘I’d do it if it was needed and China did it too’ as an opening bid, so long as we’re willing to actually ask. It’s a lot better than I would have expected – he’s taking the situation seriously.
If the Pope can help, that’s great. He seems like a great dude.
Rhetorical Innovation
As a reminder, if you’re wondering how we could possibly keep track of data centers:
A zombie challenge that refuses to go away is ‘these people couldn’t possibly believe the claims they are making about AI, if they did they would be doing something about the consequences.’
I understand why you would think that. But no. They wouldn’t. Most of these people really do believe the things they are saying about AI maybe killing everyone or disempowering humanity, and very definitely causing mass unemployment, and their answer is ‘that’s not my department.’
The originating example here is one of the most sympathetic, because (1) he is not actively building it, (2) he is indeed working in another also important department, and (3) you say having unlimited almost free high quality doctors and teachers like it’s a bad thing and assume I must mean the effect on jobs rather than the effect on everyone getting education and health care.
I do think Bill Gates, given he’s noticed for a long time that we’re all on track to die, should have pivoted (and still could pivot!) a substantial portion of his foundation towards AI existential risk and other AI impacts, as the most important use of marginal funds. But I get it, and that’s very different from when similar talk comes from someone actively working to create AGI.
Tyler Cowen clarifies (if I’m parsing this correctly) that he doesn’t think it’s crazy to think current AIs might be conscious, but that it is crazy to be confident that they are conscious, and that he strongly thinks that they are not (at least yet) conscious. I notice I continue to be super confused about consciousness (including in humans) but to the extent I am not confused I agree with Tyler here.
A good way of describing how many people are, alas, thinking we will create superintelligence and then have it all work out. Gabriel explains some reasons why that won’t work.
I think there are also major caveats on #5 unless we are dealing with a singleton. Even on the others, his explanations are good objections but I think you can go a lot farther about why these intentions are not this coherent or reliable thing people imagine, or something one can pass on without degrading quality with each iteration, and so on. And more than that, why this general ‘as long as the vibes are good the results will be good’ thing (even if you call it something else) isn’t part of the reality based community.
For your consideration:
Parmy Olson entitles her latest opinion piece on AI “AI Sometimes Deceives to Survive. Does Anybody Care?” and the answer is mostly no, people don’t care. They think it’s cute. As she points out while doing a remarkably good summary of various alignment issues given the post is in Bloomberg, even the most basic precautionary actions around transparency for frontier models are getting killed, as politicians decide that all that matters is ‘race,’ ‘market share’ and ‘beat China.’
Margaritaville
Daniel Kokotajlo is correct that ‘the superintelligent robots will do all the work and the humans will lay back and sip margaritas and reap the benefits’ expectation is not something you want to be counting on as a default. Not that it’s impossible that things could turn out that way, but it sure as hell isn’t a default.
Indeed, if this is our plan, we are all but living in what I refer to as Margaritaville – a world sufficiently doomed, where some people say there’s a woman to blame but you know it’s your own damn fault, that honestly at this point you might as well use what time you have to listen to music and enjoy some margaritas.
What’s an example of exactly that fallacy? I notice that in Rob Henderson’s quote and link here the article is called ‘how to survive AI’ which implies that without a good plan there is danger that you (or all of us) won’t, whereas the currently listed title of the piece by Tyler Cowen and Avital Balwit is actually ‘AI will change what it means to be human. Are you ready?’ with Bari Weiss calling it ‘the most important essay we have run so far on the AI revolution.’
This essay seems to exist in the strange middle ground of taking AI seriously without taking AI seriously.
I mean, yes, obviously we are helping create the tools of our own obsolescence, except that they will no longer be something we should think about as ‘tools.’ If they stay merely ‘tools of our own obsolescence’ but still ‘mere tools’ and humans do get to sit back and sip their margaritas and search for meaning and status, then this kind of essay makes sense.
As in, this essay is predicting that humans will share the planet with minds that are far superior to our own, that we will be fully economically obsolete except for actions that depend on other humans seeing that you are human and doing things as a human. But of course humans will stay fully in control and continue to command increasingly rich physical resources, and will prosper if we can only ‘find meaning.’
If you realize these other superintelligent minds probably won’t stay ‘mere tools,’ and certainly won’t do that by default, and that many people will find strong reasons to make them into (or allow them to become) something else entirely, then you also realize that no you won’t be able to spend your time sipping margaritas and playing status games that are unanchored to actual needs.
Demoralization is the central problem in the scenario in exactly the scenario Kokotajlo warns us not to expect, where superintelligent AI serves us and makes our lives physically amazing and prosperous but potentially robs us of its meaning.
But you know what? I am not worried about what to do in that scenario! At all. Because if we get to that scenario, it will contain superintelligent AIs. Those superintelligent AIs can then ‘do our homework’ to allow us to solve for meaning, however that is best done. It is a problem we can solve later.
Any problem that can be solved after superintelligence is only a problem if it runs up against limits in the laws of physics. So we’ll still have problems like ‘entropy and the heat death of the universe’ or ‘the speed of light puts most matter out of reach.’ If it’s things like ‘how does a human find a life of meaning given we are rearranging the atoms the physically possible best way we can imagine with this goal in mind?’ then rest, Neo. The answers are coming.
Whereas we cannot rest on the question of how to get to that point, and actually survive AI while remaining in control and having the atoms get rearranged for our benefit in line with goals we would endorse on reflection, and not for some other purpose, or by the result of AIs competing against each other for resources, or for some unintended maximalist goal, or to satisfy only a small group of anti-normative people, or some harmful or at least highly suboptimal ideology, or various other similar failure modes.
There is perhaps a middle ground short term problem. As in, during a transition period, there may come a time when AI is doing enough of the things that meaning is difficult to retain for many or even most people, but we have not yet gained the capabilities that will later fully solve this. That might indeed get tricky. But in the grand scheme it doesn’t worry me.
Rhetorical Lack of Innovation
It is amazing that The New York Times keeps printing things written by Cate Metz. As always, my favorite kind of terrible AI article is ‘claims that AI will never do [thing that AI already does].’
AI is already superhuman at recognizing irony, and at expressing empathy in practice in situations like doctor bedside manner. Humans ‘typically repeat or enhance what they have seen before’ or do something stupider that.
Guess who ‘the most revered A.I. researcher’ this refers to is?
The reference link for ‘studied under’ is about how Hinton was quitting Google to spend his remaining time warning about the threat of AI superintelligence killing everyone. These people really just do not care.
Beyond that, it’s like a greatest hits album of all the relevant zombie arguments, presented as if they were overwhelming rather than a joke.
If Anyone Builds It, Everyone Dies
Here is a thread with Eliezer righteously explaining, as he often does, why the latest argument that humans will survive superintelligent AI is incorrect, including linking back to another.
Is it wrong to title your book ‘If Anyone Builds It, Everyone Dies’ if you are not willing to say that if anyone builds it, 100% no matter what, everyone dies? Xlr8harder asked if Eliezer is saying p(doom | AGI) = 1, and Eliezer quite correctly pointed out that this is a rather ludicrous Isolated Demand for Rigor and book titles are short which is (one reason) why they almost never including probabilities in their predictions. Later in one part of the thread they reached sufficient clarity that xlr8harder agreed that Eliezer was not, in practice, misrepresenting his epistemic state.
The far more common response of course is to say some version of ‘by everyone dies you must mean the effect on jobs’ or ‘by everyone dies you are clearly being hyperbolic to get our attention’ and, um, no.
Here is another case from the top thread in which Eliezer is clearly super frustrated, and I strive not to talk in this way, but the fact remains that he is not wrong (conversation already in progress, you can scroll back up first for richer context but you get the idea), first some lead-in to the key line:
And here’s the line that, alas, summarizes so much of discourse that keeps happening no matter how little sense it makes:
That’s a reasonable lead-in to David Brin offering his latest ‘oh this is all very simple, you fools’ explanation of AI existential risks and loss of control risks, or what he calls the ‘Great Big AI Panic of 2025’ as if there was a panic (there isn’t) or even as much panic as there were in previous years (2023 had if anything more panic). Eliezer Yudkowsky, who he addresses later, not only is not pancing nor calling for what Brin says he is calling for, he has been raising this alarm since the 2000s.
To his great credit, Brin acknowledges that it would be quite easy to screw all of this up, and that we will be in the position of the ‘elderly grandpa with the money’ who doesn’t understand these young whippersnappers or what they are talking about, and he points out a number of the problems we will face. But he says you are all missing something simple and thus there is a clear solution, which is reciprocal accountability and the tendency of minds to be individuals combined with positive-sum interactions, so all you have to do is set up good incentives among the AIs.
And also to his credit, he has noticed that we are really dropping the ball on all this. He finds it ‘mind-boggling’ that no one is talking about ‘applying similar methods to AI’ which is an indication of both not paying close enough attention – some people are indeed thinking along similar lines – but more than that a flaw in his sci-fi thinking to expect humans to focus on that kind of answer. It is unlikely we do a dignified real attempt even at that, let alone a well-considered one, even if he was right that this would work and that it is rather obviously the right thing to investigate.
As in, even if there exist good ‘rules of the road’ that would ensure good outcomes, why would you (a sci-fi author) think our civilization would be likely to implement them? Is that what you think our track record suggests? And why would you think such rules would hold long term in a world beyond our comprehension?
The world has lots of positive-sum interactions and the most successful entities in the world do lots of positive-sum trading. That does not mean that fundamentally uncompetitive entities survive such competition and trading, or that the successful entities will have reason to cooperate and trade with you, in particular.
His second half, which is a response to Eliezer Yudkowsky, is a deeply disappointing but unsurprising series of false or irrelevant or associative attacks. It is especially disappointing to see ‘what Eliezer will never, ever be convinced of is [X], which is obviously true’ as if this was clearly about Eliezer thinking poorly and falling for ‘sci-fi cliches’ rather than a suggestion that [X] might be false or (even if [X] is true!) you might have failed to make a strong argument for it.
I can assume David Brin, and everyone else, that Eliezer has many times heard David’s core pitch here, that we can solve AI alignment and AI existential risk via Western Enlightenment values and dynamics, or ‘raising them as our children.’ Which of course are ‘cliches’ of a different sort. To which Eliezer will reply (with varying details and examples to help illustrate the point), look at the physical situation we are going to face. think about why those solutions have led to good outcomes historically, and reason out what would happen, that is not going to work. And I have yet to see an explanation for how any of this actually physically works out, that survives five minutes of thinking.
More generally: It is amazing how many people will say ‘like all technologies, AI will result or not result in [X]’ or ‘like always we can simply do [Y]’ rather than
go to therapyconsider whether that makes any physical or logical sense given how AI works, or considering whether ‘tools created by humans’ is a the correct or even a useful reference class in context.Another conversation that never makes progress:
Aligning a Smarter Than Human Intelligence is Difficult
Why can we instruct a reasoning model on how to think and have it reflected in the Chain of Thought (CoT)? Brendan seems clearly correct here.
It seems obviously useful given sufficient skill, it’s another thing you can steer and optimize for a given situation. Also it’s fun.
This works, as I understand it, not only because of optimization pressure, but also context and instructions, and because everything bleeds into everything else. Also known as, why shouldn’t this work? It’s only a question of how strong a prior there is for it to overcome in a given spot.
I also note that this is another example of a way in which one can steer models exactly because they are insufficiently optimized and capable, and are working with limited compute, parameters and data. The model doesn’t have the chops to draw all the distinctions between scenarios, as most humans also mostly don’t, thus the bleeding of all the heuristics into places they are not intended, and are not optimizing feedback. As the model gets to more capable, and becomes more of an expert and more precise, we should expect such spillover effects to shrink and fade away.
No, Guyed did not get Grok to access xAI’s internal file system, only the isolated container in which Grok is running. That’s still not great? It shouldn’t give that access, and it means you damn well better only run it in isolated containers?
Claude finds another way to tell people to watch out for [X]-maximizers, where [X] is allowed to be something less stupid than paperchips, calling this ‘non-convergent instrumental goals,’ but what those lead to is… the convergent instrumental goals.
People Are Worried About AI Killing Everyone
Joining forces with the new Pope, two Evangelical Christians write an open letter warning of the dangers of out-of-control AI and also of course the effect on jobs.
More on our new AI-concerned pope, nothing you wouldn’t already expect, and the concerns listed here are not existential.
There are two keys to saying ‘I will worry when AI can do [X]’ is to notice when AI can do [X], where often AI can already do [X] at the time of announcement.
The first is to realize when AI can indeed do [X] (again, often that is right now), and then actually worry.
The second is to pick a time when your worries can still do any good, not after that.
So, whoops all around, then.
The obvious response is ‘no, actually, pausing without being able to smell superintelligence first is (also?) not a realistic proposal, it is a fantasy.’
It seems highly plausible that the motivation for a pause will come exactly when it becomes impossible to do so, or impossible to do so without doing such immense economic damage that we effectively can’t do it. We will likely get at most a very narrow window to do this.
Thus, what we need to do now is pursue the ability to pause in the future. As in, make it technologically and physically feasible to implement a pause. That means building state capacity, ensuring transparency, researching the necessary technological implementations, laying diplomatic foundations, and so on. All of that is also a good idea for other reasons, to maintain maximum understanding and flexibility, even if we never get close to pressing such a button.
The Lighter Side
Welcome to interdimensional cable, thanks to Veo 3.
Grok decides that images of Catturd’s dead dog is where it draws the line.
Who would want that?
Oh. Right.
Also, you will soon be able to string the eight second clips together via extensions.
How it’s going.
Also how it’s going.
We don’t even have humans aligned to human preferences at home.
There is a full blog post, warning the jokes do not get funnier.
Also, did you know that You Can Just Do Math?