Quick Takes

1. It seems that Anthropic is now in the lead; they are IMO the most likely single company to automate AI R&D first. 2. I'm getting increasingly concerned about Anthropic's attitude towards alignment/safety. I grimly predict that they would basically behave like OpenBrain does in AI 2027, if they are lucky enough to get a pattern of misalignment evidence that egregious.

In Anthropic's Alignment Risk Report update for Mythos, they claim: I don't think Anthropic (or anyone) has an achievable path for keeping risk low if AI proceeds as fast as Anthropic expects (or as fast as I expect). Anthropic could (and hopefully will) take actions that significantly reduce the risk, but this won't keep risk low. More precisely: I do not think Anthropic has an achievable (>10% likely) path that results in keeping the aggregate existential risk they impose to <2%, as assessed in advance by a reasonable evaluator (the notion of risk that the risk report is supposed to estimate). Anthropic might avoid causing existential catastrophe (in fact, I happen to think this is likely), but they will impose quite a bit of risk along the way (supposing they succeed at their stated intentions of being a leading company building powerful AI within the next 5 years). My understanding is that Anthropic employees (especially Anthropic employees writing this report) often don't believe there is an achievable path to keeping risk low if Anthropic builds powerful AI / ASI in the next 5 years, so text seems incorrect or misleading. E.g., see Holden's most recent 80k episode or his post when RSP v3 came out. I wish Anthropic communicated more accurately about future risk in their risk report (the risk report is supposed to especially avoid spin). Or, if they're unwilling to do this, not commenting at all would be better. (Cross post from X/Twitter.)

re. Claude Mythos release: I'm scared.

Regarding Claude Mythos/Project Glasswing: For what it's worth, this is in line with my expectations for what LLMs should be capable of and it doesn't update me much towards their AGI-completeness. (Though it's at the more bullish end of past!me's predictions.) Receipts-wise, I'm not sure I laid it out publicly anywhere, but here's an excerpt from a message I'd sent to @habryka in January 2025: [...] I do, however, still find myself unpleasantly surprised by this. I wouldn't say it happened "earlier than I hoped", inasmuch as I did not have specific predictions about the timing... But, well, any time for it to happen would be unpleasantly early, I suppose.

I like how LessWrong now has Wikipedia preview (however over that link, it's a cute animal!). I'm going to pretend that it's because I mentioned how Gwern's window popups are part of what makes his website my favorite website design-wise and wished more websites had that. Cthulu above, it works for nature studies, science direct, substacks, and probably more!

Trying to distill why strategy-stealing doesn't work even for consequentialists: Consider a game between A and B, where at most 1 player can win and: * U_A(A wins)=3, U_A(B wins)=2, U_A(both lose)=0 * U_B(A wins)=0, U_B(B wins)=3, U_B(both lose)=0 At time 1, A has a button that if pressed, ends the game and gives 40% chance of both players losing and 60% of A winning. A can press, pass, or surrender (giving B the win). At time 2, the button passes to B, who has the same options with "press" giving 60% chance of winning to B. At time 3 if both passed, they each have 50% chance of winning. Solving this backwards, at time 2, B should press because that gives U=.6x3 vs .5x3 for passing, so at time 1, A should surrender because U_A(press)=.6x3, U_A(pass)=U_A(B presses)=.6x2, U_A(surrender)=2. In terms of theory, this can be explained by this game violating the unit-sum (mathematically equivalent to zero-sum) assumption of strategy-stealing. It confuses me that it has significant mind-share among AI safety people, e.g. @ryan_greenblatt here, despite the world in general, and technological races in particular, obviously not being zero-sum. See also my failure to "steal" the strategy of investing in AI companies.

It's widely rumored that both OpenAI and Anthropic will go public within the next year or so, triggering massive capital inflows. I think this unleashes an underrated financial dynamics. An optimal portfolio hedges your personal risks. Unless you're independently wealthy, the risk of job loss due to AI automation is one your largest personal financial risks, and it makes a lot of sense to buy insurance against that (i.e., to buy into AI companies). Right now, individual investors can't do that. And the investors that have access to OpenAI/Anthropic on the private market are precisely the ones that don't need to hedge that risk. Once you open up to retail, a rational investor should park a sizable percentage of their net worth in those shares, opening up a capital gusher. This is especially true for Anthropic -- there's basically no way for a retail investor to get meaningful Anthropic exposure if they want it. It's a little different for OpenAI given that SoftBank is kinda an OpenAI proxy at this point (albeit in ways that make it a less attractive investment than the real thing). Seems like this could lead to substantial acceleration of progress. It may also change the politics. If stopping AI progress mostly means financially soaking billionaires, then it's politically a lot more palatable than if it will also screw up the 401ks of the mass affluent.