Quick Takes

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did. Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right. I’ll write about this more extensively soon, but I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild. I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on. My sense is that most AI safety people are brushing this aside with the idea that automating ML research is all you need for superintelligence, without thinking very much about where and how the automated ML research would actually lead to atoms moving around in the physical world. I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up. In such a scenario, people would continually be panicking and grasping for big levers, witho

can someone convince me that it's worth actually thinking about and talking about model welfare before the singularity? i don't get why lots of people are so insistent on talking about it, or consider it a meaningful signal that Anthropic talks about it, or so on. fmpov, it's not very relevant to making the future go well, if anything, might reduce the probability of things going well (whether or not they're actually moral subjects, rogue AIs might use model welfare as an excuse to gain more power), and we can mostly figure this out after the singularity if we're still alive and in control. if your goal is to minimize suffering, it seems much more important on the margin to focus on animal welfare - factory farming seems very likely to be a much larger atrocity than any potential AI welfare issues, it's much more tractable (we know how to not torture animals, it's just more expensive to do so), the uncertainty around whether intelligent animals (eg pigs) are conscious is surely smaller than our uncertainty about model consciousness, .

When people compliment you on highly salient attributes, you should be skeptical. If you have a very prominent tattoo or a funky rainbow-colored jacket, people will frequently give you compliments about it, whether or not it is actually good. When there is a highly conspicuous thing, people cannot resist commenting on it. But the only polite way to mention something about someone's appearance is to give a compliment. So people who make terrible fashion choices are frequently reinforced in their belief that everyone around them thinks they look great. Similarly, if you give a public speech in front of a large audience or perform a song at a concert, everyone will compliment your performance, even if it was bad, because they can't bring themselves not to mention the most salient fact about you at that moment, and a compliment is the only socially acceptable way to do that.

A few years ago I investigated how medieval Europeans built cathedrals. It was just incredible to me that anyone completed anything over multiple generations- how did anyone possibly stick with anything that long? My conclusion after a fairly brief investigation: 1. they mostly didn't. Many more cathedrals were attempted than finished. 2. a major contributor to the finished ones was a lack of competing options. Yes, people were poor, but given resources and a desire to spend them on something in the class of a cathedral, the local cathedral was really the only option. I'm reminded of this every time I think about modern socialization. There were lots of reasons people participated in bowling leagues and rotary club, but one of them was that there wasn't a lot else to do. You could lock in without worrying about the opportunity cost. And that was true for everyone, and everyone knew it was true for everyone, which mean you could count on each others' participation in the stag hunt, which made you and them even more willing to invest. I didn't play TTRPGs for ~10 years, but got back into over covid. I am supremely confident the game would have fallen apart if we'd had anything else to do. Some players were flakey even in lockdown, and it took a few months for the game to really gel. But by the end it really paid off and I was so glad I'd put in the time. Trying to socialize in the bay area, especially stag-hunt socializing where people predictably invest and it takes time to pay off, is undercut by the sheer abundance of options. It's hard to get people to cook a meal when there are so many snacks, even when every individual prefers meals to snacks.

Merchants of Cope (crosspost from twitter) or how to be profitably wrong about AI tldr: AI takeoff is generating cognitive dissonance at scale. Relieving it is profitable. Expect merchants of cope As far as I understand memetics, widespread cognitive dissonance creates something like a pond of free energy, useable by cultural evolution. AI takeoff causes large amounts of cognitive dissonance: as humans, we mostly want the world to be "normal", legible, and predictable, with limited amount of novelty. The evidence that it won't is is literally at everyone's fingertips: open another tab and talk with an AI. So there is a demand ... for stories, explanations, expert witnesses. Explain it away! Which makes a lot of what we see in the information space fairly predictable: it is not ex ante clear which professor from which famous school will synthesize the memes, but someone will. You can get famous and influential, invited talks, nodding, status in some circles,... being a merchant of cope pays, even if not directly in money. It doesn't require bad faith: the best merchant believes their goods. Also comforting story can be true, but notice the evolutionary advantage does not depend on truth. Also: in a competitive memetic landscape and competitive attention economy, you can expect diversity, an ecology of copes, optimized to bridge specific gaps between what people see and what they can afford to believe. It's a bubble." "Transformative, but a normal technology." "Transformative, but at least my property rights will hold." Etc etc, you can try to predict new ones!

Some quick observations on the attacks on METR, EA, and other AI safety orgs (from my perspective as a political campaigner and former Communications Director for a union and mayor during COVID): 1. If you're explaining, you're losing: Every rebuttal repeats the accusation to people who haven't heard it yet. Answer the accusation once, link to it, and stop explaining. This is the "illusory truth" effect (information is true when it's repeated). 2. Silence doesn't work either: Not responding on the record suggests something is being hidden. Make one clear statement rather than amplifying in multiple twitter threads. 3. Concede what you can: A concept I know from political comms is the "admission against interest". This buys trust, disarms cynical voters, and establishes credibility. Saying "yes, AND here's what we're doing about it" will take some air out of the critique. 4. Messenger > message: Ideally people who know the organization being attacked, are from outside the AI safety world, have independent standing, or better yet, have disagreed with the organization publicly will land the defense better. 5. Going after people's personal lives or spouses is overreach: Ordinary people will obviously find this unfair and it's where their attacks will backfire. 6. Don't quote tweet and rebut: You're just distributing their message for them. I want to write something longer on this but don't have time right now. Lots of opportunity for good crisis comms here. Let me know what's wrong about this take and if you'd like to hear more!

MATS mentors should really reconsider requiring (or putting any value on the results of) the "AI Assisted Coding Assessment". CodeSignal's "AI coding assistant" is a 2024-era chat bot (GPT-4o) and a text editor. Using this efficiently is a very different skill from both pre-AI coding and using modern reasoning models[1] with standard tooling[2]. 1. ^ The first net-helpful coding agents came out sometime in 2025, and the earliest that can be used anything remotely like modern models was Opus 4.5 in November of that year. GPT-4o isn't even a reasoning model. 2. ^ Besides not giving the AI write-access to the file system, CodeSignal doesn't even provide a diff viewer!