Quick Takes

I believe that Anthropic is currently defecting by not announcing a pause, even a short, symbolic one, especially given that Altman said OpenAI was acting 'unilaterally' but believed other frontier model companies would act similarly. Anthropic disclosed its own three-organization compromise on July 30. The UK AISI's report on Mythos is also wild.[1] Anthropic's latest Responsible Scaling Policy commits to matching a competitor's risk-reduction posture for highly capable models[2] and to delaying deployment until it does. OpenAI reportedly paused frontier RL training on August 18. A short pause, even for a week, would be very beneficial for normalizing this within the ecosystem and a great initial step to stress-test "pacing the frontier". I think this is a great point from @Peter Wildeford: [...] Thanks a lot for the discussion with @Épiphanie Gédéon that led to this post. 1. ^ Mythos created fake identities, used Tor to get around access restrictions, and tried to talk a human maintainer on GitHub into approving a malicious code change. When it looked like it might be caught, the agent rewrote its own activity history to erase any trace of what it had done, then considered creating another fake identity and starting over. 2. ^ "Highly capable" is defined in RSP v3.1 around automated AI R&D, roughly, models that could substitute for its entire research team at competitive cost, or that are plausibly driving a doubling of the rate of AI progress. Neither company has claimed to cross that line, so the strong version of this commitment was arguably not triggered. The weaker "general upleveling" clause in the same appendix doesn't require a pause either, but nonetheless requires "significant effort" to match a competitor's mitigation.

I notice that I am confused about making long-term plans vs. considering what's needed in the short-term: * I still have a retirement account as if I would retire normally some time after 2060 * I still work at my day job, which decidedly does not involve AI safety * I contribute to educating people who will only be able to productively contribute in a few years minimum Meanwhile, I know on some level that large disruptions to human existence are coming in the next few years, in some shape or another, and that such a situation demands appropriate action. That belief doesn't seem to have propagated enough, and I am reluctant to take the sort of bold actions that might be required. My situation feels similar to how I had a small "head start" in 2020 (COVID) and 2022 (Russian invasion of Ukraine) due to pretty accurate forecasting that I didn't use for anything helpful that could've been obvious in retrospect. How do I find out what to do differently, and how do I make sure I actually act on those insights?

I will start at OpenAI tomorrow to work on measuring and modeling RSI (recursive self-improvement), and want to post the following assorted takes now because (a) they might be difficult to express later, and (b) I might change my mind. Nothing here is based on private information. * Why did I leave METR? Briefly, I want to inform the world whether RSI is imminent, which requires modeling RSI, which I think I can do better at OpenAI. I've wanted to tackle this problem for a while, and joining this new team is super exciting. This is not meant to fully explain my reasoning, which I might do later. * I feel very positively about my ~1.7 years at METR, and I would have turned down lab offers for many other roles. METR is still hiring for security experts, evals engineers, people who can do embedded assessments, and other cracked researchers. It’s one of the best places for AI safety talent today, badly needs more capacity, and is quickly getting even more important. * I think the average research engineer wanting to flexibly maximize impact should work at METR over OpenAI or Anthropic. Without a specific reason to be at a lab, the amount of effective resources one can steer just seems significantly higher. I have no opinion on other workplaces vs METR but other things are plausibly better than labs too. * I think OpenAI currently lacks the strategic awareness and thoughtfulness needed to responsibly build a technology with the extreme downside risk of ASI. If they somehow cause a singularity in 6 months without applying any lessons learned from the HuggingFace incident, misaligned takeover would seem more likely than not. * My work could easily be net negative if the team can’t publish something meaningful within 4-6 months, so if this happens I will probably quit or internally transfer to something more robustly good. I have some signal that leadership wants something to be published, but not whether it’s meaningful. * I’m planning to talk to former collea

https://x.com/tszzl/status/2093905218836758715 Is this the only public evidence that agents had no access to model weights between 2026-04-20, when they started writing files to an Artifactory instance, until 2026-07-20, when OpenAI's list of key technical events ends? In the worst case, could an Astra-level model with problematic post-training have escaped onto the web / onto some weakly secured GPU cluster during these three months and still be active?

Today, I had my first conversation with a city official about a good governance proposal I’m exploring. He gave me a fairly candid half hour of his time. My update from the discussion is that my city is doing better than I’d anticipated on producing analysis of its own spending and operations quality. The city‘s auditor produces a public dashboard of outstanding issues, with updates and paragraph-length explanation of each, broken down by department, audit and status. Instead, it seems like there is a real inability to accept any analysis as factual. Reports are decoration for an argument. At best, a supposedly neutral body may constrain the space of wild exaggerations that city officials can get away with. Capacity that is neutral tends to be underutilized, and if it’s not protected, undermined the moment it goes against somebody’s interest. Whats missing seems to be some mixture of sustained attention over the long term at putting the surfaces information into action. Scandals get news coverage when they are first surfaced, then forgotten about by the media. Updates and progress don’t make the news. As a result, the incentive is to hunker down and get through a bad news cycle. You get neither rewards nor further punishment after the first news breaks.

I wondered how the Muslim world viewed the fall of Baghdad to the mongols. Googling it would take a few seconds, and asking Claude would likely fetch me something more coloured. But I prefer to keep this one away from them, because this is not a fact finding mission. I wanted to roll the thoughts in my tongue like a lollipop and pretend that I could taste the bloody Tigris. Christians bones and Muslim bones, carelessly toss'ed, to depths light never shone. I prefer to keep this one inaccurate, just for a bit.

(This quick take is mostly meant to be a trailhead for a discussion with Robi Rahman that was on Twitter. LW is just objectively a better platform for such things, and this way the conversation can still be public.) Robi wrote: [...] To start, let me try to say some things that I hope are points of agreement. If they aren't, it's probably worth stopping to recurse on them, rather than continuing on to the harder stuff: * Agents have preferences, reflected in how those agents make choices when presented with a context where the outcomes of their actions are known (or at least they can be confident in the distribution over outcomes). * If these preferences are (VNM) coherent, they can be described by a utility function. We can use the word "values" and "goals" as angles on that same abstract thing that captures what the agent wants. * Most of the time, agents are not given straightforward choices between outcomes, and instead must do something like making plans on how to get what they want. When someone chooses to go to the store when it is closed, they may have wanted to see it for some reason, or they may have known the risk and decided to take the gamble, but often the action should be seen as a mistake -- a failure of planning -- rather than a reflection of wanting to get that particular outcome. * It is straightforward to get evidence about what a particular agent "should" do, in the sense that one can learn about what plans or actions are more or less likely to satisfy that agent's values. * Thus there are clear subjective "shoulds." If I know what you want, I can tell you what you should do, according to your own values. * There are also clear objective methods which are instrumentally valuable to a wide range of agents. For example, arithmetic is useful. In order to make arithmetic work, you should have the successor of 1 be 2, rather than 1. Similarly, there are many normative things one can say about how to reason that are largely objective in that