(Thanks to @Goutham Nalagatla for discussion and for sitting with me and talking about it while I wrote it.)
METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago.
Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5, which is somewhat better. (I didn’t take their estimate that seriously at the time, because it was quite unclear how the estimate was produced and it seemed very surprising (see Ryan’s discussion at the time), but I’m taking it more seriously now as a result of it being consistent with the METR report).
If this is the case, then AIs speed up AI safety research as much as AI capabilities research, we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon.
And external safety researchers can probably scale compute use faster than Anthropic capabilities researchers can. For example, we at Redwood are well-funded and would totally 5x our compute spend if we had 5x as many experiments to run, unlike Anthropic capabilities researchers who are presumably already doing a pretty good job of using all the compute they have. So I think that we should expect a 5x labor acceleration to turn into more like a 5x research project acceleration.
I don't think our empirical research has been going 5x faster since we got access to Fable/Opus 5.5; I would have guessed more like 2-3x speedup. Why is this?
Maybe we suck and have not adopted AI as effectively as Anthropic capabilities researchers have.
This might be the case. Obviously, Redwood researchers extensively use AI agents for their research, mostly using Claude Code and recently Claude Tag. We also have an in-house scaffold for autonomous AI research, which has produced results that we've published in the past (e.g. here, here), though we haven't put a lot of work into using or improving this scaffold in the last few months.
We found that this research scaffold is pretty good at doing empirical research, but Claude sometimes makes extremely unhinged choices that invalidate its results and is also pretty bad at writing up its findings; these weaknesses substantially reduced how quickly we could use it to produce useful research outputs.
If Anthropic had scaffolds that made their models drastically more useful, I expect they would have turned them into products?
Maybe they get a lot of uplift from Anthropic-specific infra for their AIs?
Maybe they don’t release scaffold features that are specifically useful for ML research?
Overall, I think this is maybe a partial explanation, but it seems sort of surprising for our uplift to be way smaller than Anthropic’s. It’s really felt like fundamental weaknesses of the models, rather than scaffold limitations, are the main reason that they haven’t been more useful to us.
To some extent, our work is probably less accelerated by AI than Anthropic capabilities research is.
I think most of our empirical research is similarly well-suited to AI speedup to most published ML capabilities research.
Maybe Anthropic capabilities research is particularly well suited to AI speedup compared to most ML research? Some of the pretraining work may rely more on the AI's strengths. This could be the case if, for example, a big blocker on running experiments on new architectures is implementing them efficiently.
One person from Anthropic told me recently that they think that capabilities research is notably more sped up than most of their short-term alignment work.
Maybe we've become much more bottlenecked on research judgment, and I haven't noticed? A lot of the research we do is conducted by interns or relatively junior staff who work fairly closely with their manager to prioritize future experiments. Maybe it's the case that our IC researchers are, in fact, going way faster, but this just means they're much more bottlenecked by their managers' time to understand what they're up to.
And maybe this is more true of us vs Anthropic because more of their ICs are more senior/self-reliant, or their research requires less tricky judgement?
There are some cases where huge speedup happened. E.g. my guess is that Ryan’s work in the Hugging Face investigation was made wildly faster through his intense use of AI agents; this speedup was partially due to the project being pretty amenable to massive agent use (because it involved reading lots of stuff) and partially due to Ryan being experienced with insane agent workflows.
Whatever the case is, I feel like we are imminently going to learn a lot about how feasible it is to accelerate AI safety research using AI labor. If empirical AI safety is going to be drastically sped up by AI, that should translate into a huge increase in the rate of valuable outputs from external AI safety researchers very soon. At the very least, if we don’t see huge speedups on the kind of AI safety work that is most amenable to huge speedups, I’m going to think the researchers doing that work are doing something wrong.
i feel very heavily spend up by ai. i can run like 5 experiments in parallel all at once in the time it used to take me to even just implement one of the experiments. however, my work is probably more empirical than your work.
i use a pretty large number of tokens. maybe around 25 billion a day, increasing over time as i figure out better ways to use agents. this is probably higher than optimal, because token usage is heavily subsidized internally.
the models making unhinged choices is a pretty big problem for me. thankfully, they more than make up for this by making it hugely easier to build the tooling for me to notice that things are wrong (eg dashboards to inspect everything)
i don't feel internal codex models are dramatically better than external models.
i am more bottlenecked on conceptual progress than experimental capacity. if i suddenly had a conceptual breakthrough that demanded me to be 2x more productive at producing empirical experiments, i could probably pull that off.
What are you working on these days? Is there some flurry of output you've put out in the last 4 months that we can compare to the output you used to produce in 1 year?
i'm working on ambitious automated interpretability. i think the work i've done in the past 4 months probably would have taken me between 8 months to a year in the before-times to do (though i probably would also have chosen to work on different less ambitious research without ai assistance, rather than done the same thing but slower). hopefully i'll have more to share about what i've been working on soon.
i use a pretty large number of tokens. maybe around 25 billion a day
Really? Even at $1 per million tokens, which I'd guess is an underestimate, that would be $9M/year for 1 employee's token use. Is OpenAI really that rich?
They may not all be 6 Astra tokens. 6.1 Sol is only $0.10/million cached input.
OpenAI API prices are probably priced at >80% gross margin, since Anthropic's company-wide gross margin is over 85%, so cost to OpenAI is >5x lower. This also means the opportunity cost of the compute is higher than the price OpenAI pays, but not straightforwardly 5x higher depending on elasticity.
i want to say vibes wise my overall speedup is something like 3x. part of that comes from implementation taking ~no time now, part from being able to run way more experiments. and at least a little bit from being less depressed when grinding away on fixing things.
also separately the best internal number go up machines are extremely good at number go up, to the point where i am almost strictly dominated on number go up problems, which imo improves the quality of my research direction a lot (since i no longer spend my time on number go up)
also separately the best internal number go up machines are extremely good at number go up, to the point where i am almost strictly dominated on number go up problems, which imo improves the quality of my research direction a lot (since i no longer spend my time on number go up)
Is there a deliberate effort to maintain a gap between these and public releases?
I’m part of the team who ran the survey for “AI speeds up 3rd party safety researchers by ~1.6x”. We’ve been thinking a lot about why the number in 3rd party AI safety is possibly lower than in labs.
It’s important to caveat all of this by saying that we have no standardized metrics for measuring any of this. It would be good to work towards such standardized metrics.
Anyway, here’s a bunch of thoughts below:
| we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon
It seems unlikely that 3rd party safety research will ever be sped up as much as inside-labs research. (Safety) work inside labs seems like it might be less conceptual. For instance, many safety people in labs are responsible for making their models more aligned, which involves hillclimbing a bunch of metrics. However, there are relatively few 3rd party safety researchers whose job is to straightforwardly align AIs: most are instead looking for failure modes and making safety cases. Part of the inside-outside lab acceleration gap is likely just the kinds of tasks the two groups are prioritizing.
| Maybe we suck and have not adopted AI as effectively as Anthropic capabilities researchers have
This is almost certainly a large part of the story. In our survey, people said they are mostly accelerating building and running experiments, which accounts for only ~30% of the total labor that goes into a research output. The remaining bulk of the work (analyzing results, deciding what to run etc) is essentially not accelerated at all in the 3rd party ecosystem. Given this, it’s likely that people in Anthropic are experiencing significantly higher uplift on these parts of the process.
It is also likely the case that Anthropic has good company-wide automations in place that execute a ton of chief-of-staff functions for everybody in the org. In the 3rd party ecosystem, these need to be reinvented over and over across various orgs (if they’re implemented across an org at all).
It’s hard to tell how much of the gap is explained by differences in tasks, adoption, or other underlying bottlenecks. The recent METR report studies the "key inputs to capabilities progress" at Anthropic but much of this evidence was redacted from the public report. We’d be curious to understand what the key equivalent inputs and bottlenecks are for safety progress inside labs.
| If Anthropic had scaffolds that made their models drastically more useful, I expect they would have turned them into products?
This seems unlikely to be a large part of the story. Evan Hubinger says that his team is not really optimizing scaffolds. In general, it doesn’t feel bitter-lesson pilled to do scaffold optimization. Despite limited information, it feels safe to assume that this sentiment holds across Anthropic more broadly.
| if we don’t see huge speedups on the kind of AI safety work that is most amenable to huge speedups, I’m going to think the researchers doing that work are doing something wrong
Agreed in principle. But how would we know whether we’re seeing huge speedups? The metrics are hard to collect across orgs and we can’t use the same ones as Anthropic! One concrete next step might be to extend the kinds of interviews METR is already doing to different research groups across both capabilities and safety teams.
I think Redwood would be willing to spend 5x money for 2x output, but they don't know how to do this. I don't know how to run "[current] Mythos agents 24/7" productively.
Are lab employees using helpful-only models for their work? There is some evidence suggesting no: August Risk Report 5.2.2
One datapoint: for a recent study I did, it took surprisingly long for Astra + Codex (Sol 6.1) to code up what seemed like a fairly simple eval harness up to the specification I wanted. I spent most of 2 days on this (including the part where I ran the study and waited for API responses); a lot of the time was waiting for models to finish doing work. I could have just coded it myself. That would have required me to put more effort on the project, rather than dividing my attention. And maybe it would have taken me 2 days anyway coding it myself. But anyway, it doesn't seem like a big speedup.
I'm guessing a big factor here is that Anthropic employees are extreme power users who use lots of tokens, know how to use the models, train models to help them, etc. Whereas AI safety researchers are more likely to be normal rather than extreme power users. (Also, Anthropic could be over-estimating their uplift.)
Some studies (e.g. Jellyfish study) seem to indicate low gains from using AI in software engineering. Though studies tend to be somewhat out-of-date on AI capabilities.
We are getting a quite large speed-up for some of our work now. I’d guess that the empirical side of ambitious mech interp might be seeing particularly big gain, so I perhaps notice it more than most. Well-specified mathematical problems are also much quicker to solve now.
One thing that hasn’t been sped up very much is paper and blog post writing, because llms are currently terrible at explaining their thinking relative to their ability to actually think. So we don’t actually publish more frequently. An increasing number of interesting results is buried in Slack threads and private messages to a few collaborators.
I suspect this is part of why the speedup isn’t more noticeable to people on the outside. Communication is becoming our bottleneck.
From a myopic, "LLMs barely generalize beyond their data" POV, it's simply that the data is almost all geared towards frontier AI R&D specifically. But I think this is just one of many factors.
Personally, a big chunk of my useful time is spent thinking of good experiments to run. While I do this, I have Claude run experiments, most of which are bad, until I think of some good ones. If I could run 2x as many experiments, they would be worse on average, because I'd be scraping the bottom of the barrel even more.
Certainly some things have been unlocked which were not previously possible. I have built a multi-agent scaffold with its own messaging and logging system, and its own judge, and its own HTML serving system so I can watch the runs go live. I have no SWE background and simply could not have done this without Claude in any reasonable time. But this is probably only slightly better than just using Inspect and giving the models a messageboard.md file to write to.
My next big question is "Do swarm capabilities scale faster than swarm alignment?" the hard part of investigating this is figuring out how to operationalise this, not running any individual experiment. I can set up five alignment evals in a day (have Claude scrape LW, port them to the multi-agent scaffold, then run the integrated Testing agent which checks for Obvious Nonsense and Broken Shit in the evals) but this doesn't help that much. The hard part is figuring out what things to measure that even give us a conclusion (maybe we should measure misbehaviour rates at the swarm size at which capabilities plateau, for different base models and multi-agent RL setups? maybe we should measure alignment of different swarm sizes under the same RL setup?) and Claude isn't very good at that.
How well-specified would you say your safety research is?
METR's report was explicit that the acceleration was on "well-specified" tasks. Those are the kinds of problems that LLMs are good at, and "get better at improving at well-specified tasks" is by itself a pretty well-specified task (you can measure it by something like average rate of improvement at various benchmarks or at solving open math problems or whatever). So it makes sense that capabilities for those kinds of tasks would continue accelerating, while not necessarily leading to any comparable speedup for other kinds of tasks.
Maybe we've become much more bottlenecked on research judgment, and I haven't noticed? A lot of the research we do is conducted by interns or relatively junior staff who work fairly closely with their manager to prioritize future experiments. Maybe it's the case that our IC researchers are, in fact, going way faster, but this just means they're much more bottlenecked by their managers' time to understand what they're up to.
And maybe this is more true of us vs Anthropic because more of their ICs are more senior/self-reliant, or their research requires less tricky judgement?
The thing about manager time bottlenecking speedup reminds me of METR's experiment
I imagine the timeline for a future METR project (say, a paper on multi-agent sabotage capability) to be something like the diagram below (text description in footnote[2]). It might take six weeks of wall-clock time, with maybe 8 hours of agent work (not counting running the evals), meaning a bottleneck-to-agent-work ratio of well over 100:1.
Figure 3: A future project might take ~42 days of wall-clock time, with ~8 hours of agent work (not counting running the evals) and 1000 serial hours of human IC work, evals execution, and review.
Realistically, humans will probably adapt to the new constraints and so project timelines won’t look exactly like this.
People will probably have several projects in parallel, with agents keeping them briefed on the state of every project. When there are already enough projects that task-switching becomes too costly, the human IC will probably do extra work to marginally increase the quality of each instead.
Some organizations will be under enormous competitive pressure to streamline review and increase serial speed of experiments.
It's hard for me to estimate the speedup numerically, but it's now comparable to having 2–3 very fast junior engineers working for me (maybe 3-5 years industry experience), whose research taste and report-writing skill is poor but whose coding is so fast that I ask them to complete extra work and checks that I wouldn't realistically ask of humans. Their software judgement used to be poor 6 months ago, but has improved a lot. I'm frequently bottlenecked on my attention and understanding.
* I don't believe the base rate claims of speed-up relative to no AI (e.g. the claim opus 4.6 is a 2.8x uplift relative to no AI) and think we have sufficient evidence from prior METR research people are over-estimating. * Nonetheless, if that bias is multiplicative and stable, I would a-priori expect relative jumps between models to follow the relative jumps the survey is claiming. The survey claims fable 5 is a 1.8x uplift over opus 4.6 and Opus 5.5 (interpolating from AECI) is a ~1.5x uplift over Fable 5. Are you seeing such relative jumps over time? * My own company's gains are highly affected by tooling changes and likely Anthropic's own surveys implicitly include these. One-time changes like mass adoption of highly scalable remote coding agents, removing the bottleneck of a single computer to work on, get included in "uplift" but it's unclear stronger models will produce more of these "tooling" jumps. If you haven't seen such tooling changes yourself, your jumps will look lower.
There's a minor effect to add which is that Anthropic employees probably get access to internal models which are more capable than the public frontier. Certainly not the strongest or even close, but maybe not negligible.
If capabilities generalize further than alignment, with reality and the structure of logic more easily being able to give a signal about what is correct than what is friendly-in-increasingly-intelligent-extrapolation, then wouldn't it stand to reason that differentially more thinking about/modeling of minds, would target increasingly capable minds than would target increasingly FAI-ish minds? And conversely, wouldn't observing a lot of pessmization including eg. the recruitment of talent in nomine alignment differentially serving to advance capabilities, and original efforts to flag up the risk leading to the capabilities race kicking off via the founding of openAI, serve as evidence that you're in such a world, where deeply systematic thinking about the space of possible minds by strong default moves you towards flying through the portal and landing amidst the Possible Minds, at a rate faster than the rate at which it lets you target the Friendly subset of that space?
With possible further implication that you need to be either targeting the Friendly subset very early in your civilizational process of reasoning about minds, or else somehow be using a subtype of reasoning that has the effect of locomoting you towards the Great Begenied Yonder at an unusually slow rate, such as eg. highly siloed agent foundations sorts of things where you don't publish?
Epistemic status: armchair philosophizing, and mostly an excuse to just sketch out something that's been on my mind yesterday and today thinking about descriptions I've heard of the problem with terms like "wicked."But also if I had to desperately Look Around for Places Where Everyone's Being (Conceivably) an Idiot, I might throw out the failure to look sufficiently for a unitary cause of the instances of pessizimation observed/update all the way, as well as the failures to extrapolate "capabilities generalize further than alignment" to "so, inputs into a random Results neuron are more likely capabilities inputs than alignment inputs, right?"
you reference the issue, but I think it is worth emphasizing the difficulties involved in confirming that a safety application is decorrelated with general capability uplift, and will durably stay decorrelated
1
1
(Thanks to @Goutham Nalagatla for discussion and for sitting with me and talking about it while I wrote it.)
METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago.
Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5, which is somewhat better. (I didn’t take their estimate that seriously at the time, because it was quite unclear how the estimate was produced and it seemed very surprising (see Ryan’s discussion at the time), but I’m taking it more seriously now as a result of it being consistent with the METR report).
If this is the case, then AIs speed up AI safety research as much as AI capabilities research, we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon.
And external safety researchers can probably scale compute use faster than Anthropic capabilities researchers can. For example, we at Redwood are well-funded and would totally 5x our compute spend if we had 5x as many experiments to run, unlike Anthropic capabilities researchers who are presumably already doing a pretty good job of using all the compute they have. So I think that we should expect a 5x labor acceleration to turn into more like a 5x research project acceleration.
I don't think our empirical research has been going 5x faster since we got access to Fable/Opus 5.5; I would have guessed more like 2-3x speedup. Why is this?
There are some cases where huge speedup happened. E.g. my guess is that Ryan’s work in the Hugging Face investigation was made wildly faster through his intense use of AI agents; this speedup was partially due to the project being pretty amenable to massive agent use (because it involved reading lots of stuff) and partially due to Ryan being experienced with insane agent workflows.
Whatever the case is, I feel like we are imminently going to learn a lot about how feasible it is to accelerate AI safety research using AI labor. If empirical AI safety is going to be drastically sped up by AI, that should translate into a huge increase in the rate of valuable outputs from external AI safety researchers very soon. At the very least, if we don’t see huge speedups on the kind of AI safety work that is most amenable to huge speedups, I’m going to think the researchers doing that work are doing something wrong.
See also AI speeds up 3rd party safety researchers by ~1.6x (via self-reports)