AI is eating Mathematics. What are we waiting for to scale up automated safety work?

Epistemic status: I spent 5 hours thinking about this in March 2026. This is a Claude-written summary of a longer document, then improved manually. This post has been on my to-do list for a while; I did the 80/20 to get this out.

I buy the core case for AI labor. There may be a brief window between early AGIs and superintelligence in which AI could do orders of magnitude more safety-relevant work than humans alone, and we are already behind in preparing for it.

Tldr: What are we waiting for? Claude just formalized Fermat’s last theorem. OpenAI released PaperBench. People are already automating mathematics. Why don't we do the same in AI Safety? Nobody is pointing compute at replicating safety papers, rerunning the fine-tunes, testing the empirical predictions we keep writing down and never checking. A lot of slop is coming, and investing in evaluating it and identifying the potential nuggets it contains is probably very important.

P.S. Since writing the initial version of the post, Resolution has been announced, but they seem to be heavily focused on alignment. This might be a good call, but this post is talking about broader safety work, not just alignment. Also, it's not nearly enough to have a single initiative working on this.

Scheming is not what I expect to break first. Writing alignment papers is much lower-stakes than automating an entire lab's codebase, which companies are already doing. The time horizon is measured in months; papers are largely independent of each other, so cumulative sabotage is unlikely, and any sabotage we catch would be useful evidence.

What breaks first is slop/evaluation. A scaled-up program produces a flood of automated research, with no reliable way to tell which parts are any good. I have run into this directly. I co-directed GRASP 2 years ago, an AI-assisted "Wikipedia of AI safety for policymakers", and we produced a large volume of AI-written content: it was very sloppy. Every single output needed a human to check it. Also, even if factually correct, readers discount anything that looks AI-generated (for good reasons: generally, something written by AI hasn't been thoroughly endorsed. The writing can be tactically brilliant and strategically very bad, with quite egregious mistakes). CURL canceled its bug bounty under a flood of AI-generated spam, even though genuine CVEs were arriving through the same channel. Scaling AI-assisted research before solving quality evaluation buys expensive slop.

I'd be interested in seeing those two benchmarks study quality evaluation:

• First: take a funder's own historical grant decisions (for small grants), give a model the proposal, web search and the relevant context, and have it produce a structured evaluation and a fund-or-reject with reasoning. Measure agreement with the human decision at each stage of the pipeline. Then measure how much human oversight, at what level of expertise, is needed to close the gap. That is a scalable oversight experiment on a task we already understand and have labeled data for. It is also immediately useful, because grantmaker capacity is a real constraint. Note: Manifund has started doing this.
• Second: reproduction of AI safety technical research. Take published empirical alignment papers, mask the quantitative results, hand the model the research question and the methodology, and see whether it reproduces the numbers. The harder version gives it only the abstract (Contamination is the obvious objection. But a model that cannot reproduce a result it has already read is not ready for the novel case, so the failure is informative either way.) Nobody (I think) has run this for safety research specifically, despite it being clearly tractable, and people are already doing this.^[1] As far as I know, the only thing preventing this from happening is the fact that we are just not starting.

2 other experiments to study the current uplift and see how far it is possible to automate today?

• A pilot measuring uplift of a team of researchers when paying per result: 10 safety researchers, 3 weeks, frontier access with the best scaffolding from the benchmarks, paid per result. Measure against what they would have produced under their normal incentives, and rerun every few months to track the trend. The quality bar should be "useful on the Alignment Forum or at a workshop", not NeurIPS.
  • Why could this be cool? Inkhaven shows that throughput shifts when incentives shift. I think tight deadlines are good. Part of what this would buy here is an exit from the academic treadmill, where researchers spend months over-polishing a useless paper because the reward is an ICML acceptance rather than impact.
  • Why not? To some extent, the karma on LW or Twitter is already a pay-per-result incentive.
• Another option could be a bounty system: an expert panel with AI-assisted screening, paying per result that clears the bar, over a public list of concrete safety engineering problems. The bounty only works if the evaluation works; we probably need to scale up evaluation infrastructure for this type of production. This enables the separation of ideas from engineering.
  • Why not? Maybe this is already how organizations work implicitly: the director proposes the idea, and the team implements it? This is potentially already similar to Requests for Proposals from big funders.

My main concerns:

• A lot of good safety research already exists and is not being absorbed, and as Buck Shlegeris put it, labs do not have the time or appetite for the 40 things that would probably solve most of the problem. Ten-xing research production without fixing the absorption and enforcement is filling a leaky bucket. But I guess we can do this in parallel.
• Solving slop is a dangerous capability externality?

1. ^{^}

   Coordinal Research was not ultimately funded.