Quick Takes

Lots of people are leaving UK AISI because the pay is too low. Often they go to work on evals at other AI safety orgs where they will get paid more money. This is terrible. It's far more valuable to have at least one government agency in the world with top tier AI expertise than it is to have a slightly stronger non-profit eval ecosystem. A competent department creates unique options for the UK government to pass a pre-deployment testing law better than other countries. Or the government could empower them as a a thoughtful AI regulator that actually understands the risks. And AISI pushes other countries to improve by setting a high bar. It could even be transformed at some point into an international body that is the backbone of an AI treaty. The jobs are in fact pretty well paid by UK tech standards but people there often have more lucrative options available at AI safety non-profits. Is there any way we can pay them to stay?

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did. Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right. I’ll write about this more extensively soon, but I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild. I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on. My sense is that most AI safety people are brushing this aside with the idea that automating ML research is all you need for superintelligence, without thinking very much about where and how the automated ML research would actually lead to atoms moving around in the physical world. I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up. In such a scenario, people would continually be panicking and grasping for big levers, witho

for my entire adult life i've been making a mistake singularity = event horizon = it is infotheoretically impossible to gain any information about the state beyond, so do not even bother trying to predict it i think dario was making this same mistake, except he projected optimism into the darkness where i projected pessimism but now the event horizon is approaching... and the interesting thing about *this* kind of singularity is, the speed of light is fuzzy. as you fall into the singularity, light from closer to the center reaches your eye. you can see more than you could previously. this is one of the really big updates i've been making over the past few months. that even *after* there exist ASI and the world is fundamentally unpredictable... I might still be around, and I will still predict that rocks will fall to the earth, and I'll probably still be correct to do so. and also that it will (perhaps) be gradual, that the uncertainty and meta-uncertainty are both on a continuum. right now, there exists humans who are smarter than I am, and yet I can still make predictions. perhaps soon there will be an AI who is twice as smart as them, and the world will be harder to predict. perhaps soon after, an AI twice as smart as that, and the world will be even harder to predict. but there probably will not be an identifiable moment where the value of prediction drops to 0. even if such a moment did exist, uncertainty about when it happens means it's still worth trying. even after strong ASI exists, perhaps!

can someone convince me that it's worth actually thinking about and talking about model welfare before the singularity? i don't get why lots of people are so insistent on talking about it, or consider it a meaningful signal that Anthropic talks about it, or so on. fmpov, it's not very relevant to making the future go well, if anything, might reduce the probability of things going well (whether or not they're actually moral subjects, rogue AIs might use model welfare as an excuse to gain more power), and we can mostly figure this out after the singularity if we're still alive and in control. if your goal is to minimize suffering, it seems much more important on the margin to focus on animal welfare - factory farming seems very likely to be a much larger atrocity than any potential AI welfare issues, it's much more tractable (we know how to not torture animals, it's just more expensive to do so), the uncertainty around whether intelligent animals (eg pigs) are conscious is surely smaller than our uncertainty about model consciousness, .

News today that Anthropic, OpenAI, SpaceXAI and Google are being sued for alleged anticompetitive behavior in agreeing to slow down development. Plaintiffs assert that "… antitrust laws do not permit competitors to decide among themselves that competition is too dangerous.“

I worked at a multinational half-owned by Accenture, as Responsible AI Lead across eight jurisdictions. In the same Twitter thread where Anthropic announced that they are partnering with Accenture, I also saw a few people asking who they are. So, for context: Accenture is one of the largest professional services and technology consultancies in the world. They have around 799,000 employees, serve roughly 9,000 clients across more than 120 countries, and generated almost $70 billion in revenue last year (See: Accenture Fact Sheet). Ocean Frogs offers a compilation here of some of their biggest clients, which include JPMorgan Chase, Unilever, Walmart, Microsoft, Pfizer, and Shell. Their reach is impressive. As to their values... I can only offer a few data points: Accenture cut more than 11,000 jobs over a period of three months last year, with CEO Julie Sweet saying they would fire people who could not be made to "embrace AI". They've also claimed that they want to become the most AI-enabled company in the world. And this month: Accenture agreed to pay the US government $25 million to settle allegations that its Federal Services business used race and sex in hiring and promotions to pursue internal demographic targets, while certifying compliance with federal anti-discrimination requirements. Accenture Federal Services, incidentally, works with every cabinet-level department in the US government and more than 30 of the largest federal organizations. Make what you will of this information.

A few years ago I investigated how medieval Europeans built cathedrals. It was just incredible to me that anyone completed anything over multiple generations- how did anyone possibly stick with anything that long? My conclusion after a fairly brief investigation: 1. they mostly didn't. Many more cathedrals were attempted than finished. 2. a major contributor to the finished ones was a lack of competing options. Yes, people were poor, but given resources and a desire to spend them on something in the class of a cathedral, the local cathedral was really the only option. I'm reminded of this every time I think about modern socialization. There were lots of reasons people participated in bowling leagues and rotary club, but one of them was that there wasn't a lot else to do. You could lock in without worrying about the opportunity cost. And that was true for everyone, and everyone knew it was true for everyone, which mean you could count on each others' participation in the stag hunt, which made you and them even more willing to invest. I didn't play TTRPGs for ~10 years, but got back into over covid. I am supremely confident the game would have fallen apart if we'd had anything else to do. Some players were flakey even in lockdown, and it took a few months for the game to really gel. But by the end it really paid off and I was so glad I'd put in the time. Trying to socialize in the bay area, especially stag-hunt socializing where people predictably invest and it takes time to pay off, is undercut by the sheer abundance of options. It's hard to get people to cook a meal when there are so many snacks, even when every individual prefers meals to snacks.