Quick Takes

I am pretty annoyed with the messaging around METR's latest result. Our chief concern is RSI dynamics, i. e., AI capability progress making subsequent AI capability progress easier. The way it would show up on graphs is by making straight lines steeper and steeper. If we observed 1.5x acceleration in one time period, we should observe 2x acceleration in the next time period, 2.5x in some subsequent one, et cetera. This is what RSI implies: a line continually going up on the graph of productivity multipliers. In other words, research acceleration at T+1 should be a function of research acceleration at T. This is what scenarios like AI 2027 assume: that there are compounding multiplicative dynamics at play. But that is not what the graph in the report, if taken at face value, shows. The graph shows a single slope change around the start of 2026 – something that made Anthropic permanently better at AI research – and then no additional productivity improvements all throughout 2026. This is better modeled as a one-off "productivity unlock", rather than smoothly predictable RSI.[1] In particular, it's not straightforwardly extrapolable into claims of the form "we will reach 4x acceleration by January 2027, 8x acceleration by March", etc. Taken at face value, the graph says we reached 2x acceleration by early 2026, and that we're still at 2x acceleration. Will we stay at 2x all throughout 2027 as well? Will there be a different discontinuous slope change that puts us at 10x next week?[2] Who knows. There are two model breaks: (1) the current speed-up to AI research from AI is much larger than previously predicted, (2) the current speed-up to the speed-up is much lower than was expected at this speed-up value. This means that whatever this 2x-speedup figure is, it's not the same type of figure as the compounding multiplier in AI 2027. (Say, maybe the straight line on the multipliers graph should have a vertical offset, because significant aspects of the speed-up are non

I struggle to believe at an intuitive level that autonomous R&D is so ingenious. I have been monitoring a few side projects: 1. Building a new SOTA engine for Scrabble (current SOTA is not really a neural-network method and doesn't beat the strongest human player) 2. Building a bot to play Ultimate Tic-Tac-Toe 3. Researching philosophy and formalizing some proofs for the AI Philosophy contest In all of these cases, the model's judgment is just very poor in a basic way. For example: 1. When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful. 2. It will conduct experiments that have no information value because the amount of training data used was obviously too small to distinguish the architectures to be compared. 3. In the philosophy project, where it built a wiki over many tokens and investigations, it kept going down directions which were extremely trivial and had little value. 4. For Ultimate Tic-Tac-Toe, I saw a very clever idea in a blog post that I doubt Astra would ever have come up with; it actually located the blog post but didn't realize its importance or relevance (which could be discerned easily from his strong performance in an online arena and the recency of the post) over papers from previous years that are obviously less relevant. I judged an Apart hackathon last weekend and reviewed eight papers, all of which scored between 60% and 95% on Pangram. Predictably, the papers were often very similar to each other, and kind of methodologically incomplete in a vague but similar way, where they seem to overlook the most straightforward analysis. ---------------------------------------- Maybe the important exception is certain hill-climbing tasks with fast feedback loops. For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every me

The recent METR report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027. I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind. I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust. Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements. A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 3.8x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027. Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger: * Not

Pacing shouldn't use risk assessment.* Even if you're magically perfect at risk assessment. Rather than restricting creating new models, it should involve restricting capabilities research. (And you have to restrict it everywhere, so that e.g. Microsoft doesn't turn into OpenAI's capabilities research shop.) Here are some examples of pacing which could actually be bad: * Restricting/delaying/canceling risky external deployments * External deployments are good because they enable visibility of capabilities and safety research. (And, less importantly, diffusing capabilities might decrease the future chaos when much more capable models are released.) (But there should be restrictions aimed to prevent people from using it for AI R&D.) * Restricting/delaying/canceling risky internal stuff (pretraining, posttraining, maybe internal deployment[1]) without restricting experiments/etc. * This regime is likely to be weakened/ended before superintelligence. For one, the research will likely leak or be stolen by foreign actors, and then US actors will feel strongly incentivized to progress quickly. And when unrestricted progress resumes, it will go faster than it would have by default, since there will be a large overhang of capabilities research to use in the next model.  * Risk-assessment-based restrictions (on pretraining, posttraining, internal deployment, whatever) * Same as previous bullet. * Standards that look like no scary models before safety/security milestone X * Same as previous — the company will delay the scary model but keep making algorithmic progress. The main benefit of pacing isn't preventing risk from the next model — it's having more time before reaching superhuman AI researchers. The company isn't imposing catastrophic risk yet isn't a crux; if you only pause right before you're directly imposing lots of risk, that pause will be very unstable because others can steal your research. Ways to restrict capabilities research include (1) re

The LW homepage algorithm should place more weight on recency (relative to karma), for highly active users. If a user has <2500 karma, often when they publish a post, it will get literally zero time on the homepage. And so ~nobody will see it. Probably some of these posts are good but since they never appear on the homepage they never get to experience the virtuous cycle of getting upvoted ⇔ more people reading them. Meanwhile when I go to lesswrong.com, I mostly see posts that either I've already read or I've already decided not to read, since I go to lesswrong.com multiple times per day. It would be slightly better for me if the algorithm put more weight on recency when showing me posts, and it would be better for others because I'd sometimes read good recent-low-karma posts and upvote them so others could see them! And I sometimes feel discouraged about posting because I think maybe I'll write a post people-would-like-if-they-saw but nobody will see it. The solutions are bad: * Sometimes I write things-that-should-be-posts as shortforms because it's easier for shortforms to be seen. (But I think the shortform algorithm should also be tweaked, similarly to the post algorithm.) * Sometimes when I publish something and the homepage-bar that day is too high for a brand-new 9-karma post to be shown (or it drops off the homepage after just a few minutes, before experiencing the virtuous cycle) I get a friend to upvote it. It's bad that I have to do that! And often those posts then do well; it's not like the algorithm is correctly predicting that the post won't do well. And I'm privileged to have my posts start with 9 karma — it's way worse for people whose posts start at ≤6 karma! (I've DMed the LW team about this several times, to no avail, but maybe if I post here it'll work. I feel like this might be a super easy way to make LW like 5% better — partially from showing-people-better-posts and partially from making authors feel better about writing posts.)

Here are a few claims I believe:  1. AI companies will probably face a choice in the next 0-2 years to either (i) incur a significant risk of AI takeover or (ii) slow down capabilities progress.  2. If an AI company slows AI capabilities without slowing down algorithmic progress[1], then they will accumulate unused algorithmic progress (which I’ll call an algorithmic overhang). 3. Algorithmic overhangs are a major downside, sufficient to make me uncertain about whether slowdown proposals that incur algorithmic overhangs are net-positive.  4. The good versions of safety cases or limitations on model capabilities work via limiting algorithmic progress.  5. The two most promising ways to limit algorithmic progress are compute reallocation and internal deployment lags (Options 2&3 here).  1. Compute reallocation: limiting the amount of compute that can be spent on applications that increase capabilities (e.g. experiments, training runs, etc).  2. Internal deployment lags: limiting the capability of models used on AI R&D (for example, requiring a six-month delay before you can use a newly developed model on AI R&D).  AI companies and governments should be implementing policies that limit the generation of new algorithmic progress (such as those in Claim 5), as opposed to the capabilities of the final training run. I’m very concerned that by default most people designing AI policies do not understand these considerations, and so will end up with policies that are net-negative or much worse than they could have been.  Below I’ll go into a little more detail on claims 2, 3, and 4.  Claim 2. If an AI company slows AI capabilities without slowing down algorithmic progress, then they will accumulate unused algorithmic progress.  There are several proposals for slowing down that don’t significantly delay algorithmic progress:  * Limits on benchmarks or benchmark aggregates (e.g. ECI or ECI / year)   * Limits on pretraining validation loss  * Pacing due to

My takeaways from METR's Independent Assessment of AI R&D Acceleration at Anthropic: 1: Anthropic and OpenAI employees were right about timelines (or at least closer to right than the general LessWrong consensus). Timelines seem short. (I haven't had time to figure out how short yet but it seems like things are going faster than AI 2027.) Anthropic announced the same 2x acceleration number as METR in their S-1 and nobody took it that seriously because people (myself included) thought Anthropic tended to overhype things or be in a bubble. But they were just exactly right. The information in the METR report makes timelines seem short because... 1.A: The best models are somewhat better than we thought, Mythos wasn't a one-off jump in capabilities. You might have thought the acceleration was more like the green or even the purple line, but it's faster than both. 1.B. "We guess that a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI. " I think many people thought the acceleration that got us to Mythos was from things like increased compute, not RSI feedback loops. 1.C: The benchmark scores are not overfit/gamed: METR looked at many sources of evidence to reach their conclusions, specifically: [...] and not only was it all consistent with 2x acceleration, if anything "some of this evidence indicated that acceleration could be above 2x and increasing over time". Reading the report made me reflect on how much clarity there is around the extra information folks inside AI companies have. I think I hadn't priced in that having all these clear benchmarks with new models might make it somewhat difficult to dramatically overestimate progress that's a few years out. 2: METR didn't identify any mitigating factor that might lengthen timelines; "We did not identify any convincing bottlenecks to continued rapid progress." I keep trying to make up stories for why timelines will be longer: maybe the fuzzy long-horizon tasks won't be automated, m