Quick Takes

Jacob Coxon’s resignation tweet currently has 120M views in less than 24 hours. Astra thinks the 235k bookmarks - 36 for every 100 likes - is an extraordinary level of engagement, of people wanting to come back to the thread. And Evan Hubinger’s reply is at 35M views. This might be one of the most important AI risk messages to date. Edit: for reference, Grok says it’s in a similar class to “Strong Elon Musk posts, major celebrity / death / personal news, breaking tech / AI / industry bombshells, viral videos from big creators, political / cultural flashpoints, and well-timed news from mid-to-large accounts”

When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle. I don't think we're in the crunch time. ---------------------------------------- All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, but it wants to stay as close to lookup tables as possible. What can we say about its trajectory? AI progress is driven by parameter scaling, algorithmic improvements, data improvements, and RLVR. Of those: * Parameter scaling improves capabilities generally: larger models are in fact better at ~everything, including various "illegible" skills. However, this type of scaling can't be pushed much farther much faster, and another general Opus 3 -> Fable 5 jump is not going to birth a strategic supergenius.[1] * Algorithmic improvements are not really a big contributor to things, see this and this. * Data improvements basically means hiring skilled human labor to generate annotated data traces. This also doesn't really scale much farther much faster, and there are no runaway loops possible this way: all capabilities are improved and encoded in a basically "manual" manner. * RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains, and may even degrade scarier general-purpose capabilities. Now, there are a bunch of counter-points one may raise here: * "Aha, but DL research is itself an easily verifiable domain!" * Sure. DL research can produce a model that is really good at doing the sort of thing that DL researchers are doing right now, improving models in the ways they

Anthropic pretraining researcher resigns over safety concerns: https://x.com/hilbertspaess/status/2097476196791709843 [...] If you work at an AI lab, possibly the most impactful you can do now is resign and try to start a social cascade that stigmatises participating in the AI race. But please do it right. Most AI employee resignations so far have been unstrategic, seemingly spur of the moment decisions with little consideration about making the most of the opportunity. You should have a media strategy lined up before you leave, journalists briefed, statements written and edited. Or even better, you should not directly resign but instead use your newfound freedom to send the costliest possible signal to your company and to the world that you are actually scared about AI - to the point where you are less worried about keeping your job than doing something to slow AI progress. If you are in this position, DM me if you want help executing this as effectively as possible. Edit: It seems that Jacob Coxon did have a media strategy and it paid off.

I feel pretty worried that this is the calm before the storm. I think starting with late 2026, we are officially in something like "crunch time." I concretely mean that it seems like things are really speeding up and the stakes are getting higher. * We have to treat our multiple-month research projects in the mindset of "this is likely one of the last research projects I ever do before this kind of work is automated by AIs." * The kinds of engagements, setups, and precedents that are set by labs and third parties right now will substantially dictate the shape of takeoff. The time to pilot whatever things we want during takeoff is right now. We can't do it later. If we want to do major pacing at any point, we need to start laying the foundations right now. We just don't have much time to get that many reps in. * Relatedly: 'I think pausing/pacing the frontier is less of "an action that you can only do once and thus have to conserve until dangerously late into AI takeoff" and more of a hazy set of actions, all of which can be practiced, and most of which get easier if you have earlier precedent for them. It seems really productive to set up systems right now where labs can take net-positive actions and verify them to each other and the rest of the world. We can get practice with the small versions before we can achieve the large versions.' * As more high-profile incidents happen (and possibly, as the first casualties mount), there will be wild and sudden waves of pressure, both internal and external to the labs. These waves could be extremely counterproductive or extremely productive based on what sorts of asks and changes they materialize. * There will be so much more politics everywhere. Politics in the naive sense, and politics in the "balancing fragile and high-stakes relationships rather than object-level things." * Capabilities will just. Keep. Advancing. The models will be able to do more and more things. Various thresholds I have personally set for "

It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization.

You cannot fix the any of the fundamental causes of misalignment by patching broken RL environments. An RL environment being "broken" is merely one that exposes much more pathological-looking unblocked strategies to the agents being trained in it, but this is a difference in degree rather than kind. Have you noticed how coding agents tend to write pretty janky code, when it comes to their coding style? What about Claudish? Many of the tendencies that are reinforced by RL environments (and various other factors like length penalties, etc) produce behavior that many people consider undesirable. This is just "we don't know how to specify the True Reward Function"[1] all over again. RL environments aren't anything special; they're just demonstrating (again) that we don't have a regime for post-training meaningful capabilities into models without substantially changing their behavior, by way of their values/learned propensities/etc. 1. ^ Outer alignment. Let's not even talk about inner alignment.

Tristan Buckmaster, a mathematician at NYU's Courant Institute, says OpenAI researchers told him that an internal model had produced a proof of finite-time blowup for the Navier-Stokes equations with smooth forcing. If correct, this would resolve the Millennium Prize problem in the negative. Buckmaster and his collaborator Levent Alpöge (who works at Anthropic) had spent a year working on this problem, with the help of Claude and Codex, and had released Lean-verified proofs for related problems. He was concerned that OpenAI might have been training on their Codex sessions. He described a call with Sébastien Bubeck and another OpenAI mathematician as follows: [...] This is pretty egregiously bad behavior from OpenAI IMO. This is not how a frontier lab should be interacting with the math community.