Quick Takes

I struggle to believe at an intuitive level that autonomous R&D is so ingenious. I have been monitoring a few side projects: 1. Building a new SOTA engine for Scrabble (current SOTA is not really a neural-network method and doesn't beat the strongest human player) 2. Building a bot to play Ultimate Tic-Tac-Toe 3. Researching philosophy and formalizing some proofs for the AI Philosophy contest In all of these cases, the model's judgment is just very poor in a basic way. For example: 1. When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful. 2. It will conduct experiments that have no information value because the amount of training data used was obviously too small to distinguish the architectures to be compared. 3. In the philosophy project, where it built a wiki over many tokens and investigations, it kept going down directions which were extremely trivial and had little value. 4. For Ultimate Tic-Tac-Toe, I saw a very clever idea in a blog post that I doubt Astra would ever have come up with; it actually located the blog post but didn't realize its importance or relevance (which could be discerned easily from his strong performance in an online arena and the recency of the post) over papers from previous years that are obviously less relevant. I judged an Apart hackathon last weekend and reviewed eight papers, all of which scored between 60% and 95% on Pangram. Predictably, the papers were often very similar to each other, and kind of methodologically incomplete in a vague but similar way, where they seem to overlook the most straightforward analysis. ---------------------------------------- Maybe the important exception is certain hill-climbing tasks with fast feedback loops. For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every me

The recent METR report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027. I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind. I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust. Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements. A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 3.8x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027. Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger: * Not

Here are a few claims I believe:  1. AI companies will probably face a choice in the next 0-2 years to either (i) incur a significant risk of AI takeover or (ii) slow down capabilities progress.  2. If an AI company slows AI capabilities without slowing down algorithmic progress[1], then they will accumulate unused algorithmic progress (which I’ll call an algorithmic overhang). 3. Algorithmic overhangs are a major downside, sufficient to make me uncertain about whether slowdown proposals that incur algorithmic overhangs are net-positive.  4. The good versions of safety cases or limitations on model capabilities work via limiting algorithmic progress.  5. The two most promising ways to limit algorithmic progress are compute reallocation and internal deployment lags (Options 2&3 here).  1. Compute reallocation: limiting the amount of compute that can be spent on applications that increase capabilities (e.g. experiments, training runs, etc).  2. Internal deployment lags: limiting the capability of models used on AI R&D (for example, requiring a six-month delay before you can use a newly developed model on AI R&D).  AI companies and governments should be implementing policies that limit the generation of new algorithmic progress (such as those in Claim 5), as opposed to the capabilities of the final training run. I’m very concerned that by default most people designing AI policies do not understand these considerations, and so will end up with policies that are net-negative or much worse than they could have been.  Below I’ll go into a little more detail on claims 2, 3, and 4.  Claim 2. If an AI company slows AI capabilities without slowing down algorithmic progress, then they will accumulate unused algorithmic progress.  There are several proposals for slowing down that don’t significantly delay algorithmic progress:  * Limits on benchmarks or benchmark aggregates (e.g. ECI or ECI / year)   * Limits on pretraining validation loss  * Pacing due to

My takeaways from METR's Independent Assessment of AI R&D Acceleration at Anthropic: 1: Anthropic and OpenAI employees were right about timelines (or at least closer to right than the general LessWrong consensus). Timelines seem short. (I haven't had time to figure out how short yet but it seems like things are going faster than AI 2027.) Anthropic announced the same 2x acceleration number as METR in their S-1 and nobody took it that seriously because people (myself included) thought Anthropic tended to overhype things or be in a bubble. But they were just exactly right. The information in the METR report makes timelines seem short because... 1.A: The best models are somewhat better than we thought, Mythos wasn't a one-off jump in capabilities. You might have thought the acceleration was more like the green or even the purple line, but it's faster than both. 1.B. "We guess that a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI. " I think many people thought the acceleration that got us to Mythos was from things like increased compute, not RSI feedback loops. 1.C: The benchmark scores are not overfit/gamed: METR looked at many sources of evidence to reach their conclusions, specifically: [...] and not only was it all consistent with 2x acceleration, if anything "some of this evidence indicated that acceleration could be above 2x and increasing over time". Reading the report made me reflect on how much clarity there is around the extra information folks inside AI companies have. I think I hadn't priced in that having all these clear benchmarks with new models might make it somewhat difficult to dramatically overestimate progress that's a few years out. 2: METR didn't identify any mitigating factor that might lengthen timelines; "We did not identify any convincing bottlenecks to continued rapid progress." I keep trying to make up stories for why timelines will be longer: maybe the fuzzy long-horizon tasks won't be automated, m

I think of November 4th as Rotblat Day. Joseph Rotblat was the only scientist to leave the Manhattan Project on moral grounds. I have a little “Rotblat doc” inspired by him, where I’ve written down what motivates me and what might cause me to leave a job (or seriously consider it). I think of this as a way to avoid getting swept up or drifting over time in ways I don’t endorse. [1] At the moment, though, I get even more out of his later work against nuclear weapons and war -- especially what others described as his “enduring optimism” and “pragmatic idealism” in the face of catastrophe. In his own words: [...] and: [...] ---------------------------------------- More in a post I wrote about him. (I also really appreciated this more recent post.) source 1. ^ The idea of keeping a doc like this isn’t mine.

Catastrophic Talent Misallocation The central talent problem in AI safety is not “making the number of full-time people in AI safety go up” but “supporting niche talent needs”. It is becoming clearer what work lies on the critical path, and the majority of people in AI safety are not doing it. Rationale: 1. The situation is urgent. Note that this is not necessarily the same as timelines being short, but it is similar. The political winds are shifting (perhaps in our favor) while AI becomes a partisan issue and labs are getting closer to RSI.[1] 2. There is a massive misbalance in the number of people working in certain subfields relative to their importance. The number of aligned people working in comms is probably in the double digits. There is at least 10x that amount of people working on technical research, and I don’t think that comms is less than 10% as important as technical research. Of course, not all of the technical researchers would be great at comms, but I find it hard to believe that less than 10% of them would be decent, especially if put on the right team. 3. We don’t need safety engineers at labs papering over cracks.[2] However, the fact that they have crack-papering experience is valuable in speaking credibly about the risks, whether to DC people or mass audiences. 4. I suspect that many current employees of METR or Redwood are smart enough that they could tackle a new vertical (e.g. operationalizing what a kill switch looks like) relatively soon, but I find it harder to believe someone new to AI safety could do the same successfully, because they lack connections and general AI risk know-how. 5. The first comms and generalist fellowships (mostly) happened this year. There have been over 10 iterations of MATS, which has churned out great results, but there are a ton more MATS alumni than Generator alumni. Counterarguments: 1. It’s good to not put all your eggs in one basket. My counter-counterargument is that even if all the fieldbui

Tyler Cowen has a new free book about the history of marginalism and how ideas arise generally, and something something AI something (I don't particularly expect to find what Cowen has to say about AI that interesting). Early on, he draws a contrast between intuitive marginalism (the usual intuitive application as seen in those trained in economics discussing an application) and tautological marginalism, where for example someone apparently not calculating decisions at the margin is seen as believing the marginal utility of calculating further is lower than the marginal cost of thinking longer. Those who have read some of the posts on coherence/selection theorems may know of a similar issue - we can interpret a rock as having 'preferences' over histories and putting a high utility on whatever it actually does. Cowen states: [...] This is essentially the same stipulation as what I've seen John Wentworth add: there is an implicit hypothesis that the preferences are over some, but not all, of the variables, and that what we observe is a consequence of having preferences that are "sparse" over the world in some sense, or "factorize well", or something like that. Intuitively, highly detailed preferences over the universe history that cannot be factorized into caring about variables like "money" or even "fun experience-moments" are not something actual agents are likely to have - at the very least, they have relatively small minds as compared to the world.