I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathematical models of the outcome distribution, or of where the marginal dollar stops being highly effective.
AI has been a godsend for all of this. It can research, model, and - as of this year - superforecast outcomes. It can do what would be hours of work for a human in minutes. I found myself using it more and more. But however much I used it, it always felt like I was the bottleneck. The parts I could outsource to AI were sparkling, perfect, exhaustive - but they were all tacked together with shoddy connective tissue formed by the Mark 1.0 human brain. So I wondered if I could automate grantmaking.
I started with the dumbest possible strategy. I assigned the best thing I could think of - a well-crafted US-China pause based on the AI Futures Project's Plan A - an arbitrary value of one million points.[1] Then I generated about eighty lesser AI safety milestones - from MATS to HPMOR to the RAISE Act to the emergent misalignment paper - and subjectively assigned them some number of points based on how good they seemed relative to Plan A and each other. Then I fed it to the AI to give it a sense for how good a "point" was in this framework. Then I asked it to forecast how many points various grant proposals were worth. The system has since accrued lots of extra epicycles and features and complications, but that's still the skeleton.
As far as I can tell, this dumbest possible strategy worked well. The autograntor gives common-sensical results that match my impressions and the vague gestalt funding order of Lightcone grantors. Its reports surface all relevant information, and its reasoning is clear. The single highest cost-effectiveness grant it's found so far is also the one that got the most interest during the last Lightcone run.[2] When I tested it on a bottom-tier proposal that no human evaluator even considered, it gave cost-effectiveness numbers two orders of magnitude worse than the genuinely-under-consideration grants that I had tested it on before.
This post will present details for people who want to examine these ideas or build on them further. The current version of the autograntor has four parts: a Milestones document, a FutureSearch connector, a PERCYs module, and a Common Law. I'll start by reviewing each, then give some sample grant analyses, then attach the files you need to run the autograntor yourself.
The Milestones
The milestones are the earliest place that this strategy could fail. Can I really translate my beliefs about what matters in AI safety into a list of milestones, each with a consistent point value?
This combines two questions. First, can I, Scott, generate a set of milestone point values that accurately and consistently conveys my own worldview? Second, if someone else were to translate their own worldview into milestones, would it agree with mine?
I chose to deprioritize the second question. Yes, I have different beliefs about AI safety than other knowledgeable people, but this is an inherent problem for my grantmaking, even if I did it manually. If we could solve the first problem, then I could be confident that the autograntor was correctly representing my opinions, and someone else who wanted to use an autograntor could confidently ask it to represent their opinions instead.
To investigate the first question, I asked four AIs - Opus, Fable, ChatGPT, and Grok - to guess how I, Scott, would fill them in.[3] If all four got similar results, then "Scott's worldview" constrains the point values well; if all of our results were very different, then noise in the translation of a worldview into numbers overwhelms signal.
Two-way correlations ranged from 0.95 - 0.99, which might sound extreme, but some of the milestones were obviously many orders of magnitude better than other milestones - it's not hard to tell that achieving global pause with China is much better than successfully running the Manifest conference.
A better measure is the geometric-mean difference between two AIs across all milestones. This ranged from 1.6x (Opus vs. ChatGPT) to 2.6x (ChatGPT vs. Grok).
Sample table of AI milestone predictions
Since my true milestone value list came from taking the average of many different methods of calculation, the real variance is something like the square root of these numbers, ie from 1.3x to 1.6x. I consider this good enough to work with.[4]
I would have liked to have gotten some humans to try to predict my point value within my world-view, but couldn't get anyone to agree to do this (except for one person who agreed but used an AI, unintentionally duplicating my procedure).
Although I don't think the second question (difference between people with different worldviews) matters for the success of the autograntor, I did investigate it via a survey where I asked people to fill in their own values. Nine people agreed, including four people whose names I recognized and who I expected to have good models (other people might also have good models, I just didn't recognize their names). The geometric mean difference between myself and those four people was 2.9x, 4.4x, 6.9x, and 8x, so much larger than the difference between different AIs trying to estimate my view. Outside that artificial limitation, geometric means were even higher - for example, one pair had a geomean difference of 15x! To give an example of where such differences came from, "Harry Potter And The Methods Of Rationality" was an especially controversial milestone - one person rated it as 1 point, another person as 75,000 points.[5]
Everything below assumes that the Milestones list is good enough to work with.
The FutureSearch Connector
FutureSearch is an AI superforecaster. In last season's Metaculus Cup, it took third place among bots, and separately would have taken third place among humans if it had been human. I think these differences are mostly noise, and it should be considered approximately as good as the top human superforecasters. I wanted to take advantage of this skill to predict grant outcomes.
My original plan was to run the autograntor entirely on FutureSearch. This failed badly. FutureSearch wasn't designed to do things more complex than a simple clearly-framed forecast, so it bungled the various calculations involved in the autograntor process.
What seems to have worked is to run the autograntor on Claude Opus, but give it a connector that allows it to call FutureSearch via the API for all components of the grant analysis that benefit from a probabilistic forecast. For example, in one case where the success of an organization depended on it getting a new CEO quickly, Claude gave the relevant information to FutureSearch and asked it to predict the probability that it would find a new CEO within six months. In another case where an organization was working on verification technology, Claude asked FutureSearch to predict whether the political situation would lead to a treaty where the relevant verification was necessary.
FutureSearch has kindly given me a free subscription, but if you're paying through the API, it's about $1 - $5 per query like this. I asked Claude to limit itself to five queries per grant analysis.
Preseen and Mantic have competing AI superforecaster products with equal evidence of accuracy. I'm using FutureSearch because I'm most familiar with it, it seems to be the fastest (unless there are ways to speed up the others that I don't know about), and it seems to have the most convenient connector to Claude (unless the others have connectors I haven't noticed yet). Future autograntor models might explore the advantages of these other products, or try to aggregate forecasts from all of them.
PERCYs
Many grants' theory of change focused on persuasion. There were a wide variety of persuasion grant types, from PLZDONTKILLUS (ordinary people watch short videos and become more informed/concerned about AI) to Palisade (demos to Congressmen make them more informed/concerned about AI).
Rather than force this into the milestone system, I created a unit of persuasion, the PERCY (Persuasion-Year), equal to flipping one average person from neutral about AI risk to concerned about AI risk, for one year. Through a process similar to the Milestones, I then tried to assign multipliers to various factors - what if the person is a journalist? a Congressman? a frontier company employee? Xi Jinping? what if instead of just making them concerned, it turns them into a dedicated activist?
This system started as another attempt to convert AI superforecasters' predictive ability into grantmaking ability. The superforecaster can predict that a certain channel we're considering a grant to will get 500,000 viewers, of which 3 will be Congressional staffers and 6 will work at frontier labs, and then I can turn that prediction into a step in a grant evaluation.
There are currently some serious flaws in this scheme - most notably, every method of adjusting for double-counting (the fact that two properties might be trying to persuade the same audience) fails in a different way. I'm currently banking on these absolute failures still allowing for relative comparison - ie the results when comparing different grants are still somewhat sensible. As a stopgap, in some of the documentation I've replaced the absolute measure of PERCYs (which is a probably-false claim about the absolute number of people persuaded) with a relative measure, "miles per hour" (a comparison to the number of people persuaded by one hour of Rob Miles video content).
One of the Milestones on my list is the value of a PERCY. I currently rank 1000 PERCYs = about 1 point.[6]
The Autograntor Common Law
Autograntor analyses often produce difficult questions of interpretation. For example:
Suppose that I'm considering two organizations. One can generate 1,000 points of value with my money alone. The other can generate 1,000 points of value by taking my money, and also consuming the time of one hundred activists. It seems like the second is worse, because the cost is the opportunity cost of those activists' time. How should that be factored into the analysis?
Suppose that a grant will produce 1 point of value per year. Naively, if the universe lasts forever, its value is infinite. In practice, we should either cut it off or apply some discount rate. How do we do that?
Suppose the autograntor finds evidence that another funder is considering granting to an organization. Should it act as if the organization already has the money? Doesn't have the money? Has the money, with some probability equal to the probability that the other funder will actually come through?
Suppose I am considering funding half of an alignment researcher's salary, with another funder providing the other half. Should I assign my grant half-credit for their output? If the alignment researcher only got half the expected salary, wouldn't they either prioritize the most important half of their work, or grudgingly do all of it and live on less money than expected? But if both I and the other funder think this way, we will underestimate our total contribution.
At first, I despaired of ever solving all of these. Later...
...I decided to hand them off to the AI as they came up. I told the AI to make its best judgment according to the spirit of the exercise and record its ruling in a Common Law document. Later autograntor runs would respect previous rulings for the big questions and add their own sub-rulings for smaller sub-questions. Every so often, I would review the Common Law and make sure everything in it was sensible and matched the spirit of the exercise. This ensures that at the very least, all grants will be judged according to similar criteria.
The current Common Law is up to about a hundred rulings. Predictably, I'm delinquent in my resolution to review it regularly.
The Prompt
Here's how I start the autograntor:
I would like your help with AI safety grantmaking. In particular, I would like you to superforecast the outcome of the following grant:
...measured in the utility point units defined. In the project documents folder, please find:
a canonical list of milestones and their utility weights. There is another file of AI-generated provisional milestones if you need more.
a description of PERCYs, a separate module of the model used for persuasion/communication grants. You don't have to consult this if you don't think it's relevant. If you do use it, use the PERCYs-to-utility conversion on the main milestones doc.
The "Autograntor Common Law", previous AI and human users' decisions on how to resolve various edge cases. You don't have to consult this if you don't run into any edge cases.
Please forecast how much extra utility (based on the milestones doc) the grant will provide according to the grading criterion below, at the following amounts:
[POTENTIAL DOLLAR AMOUNTS FOR GRANT]
If I forgot to provide any of the documents, or to fill in the grant amounts, please remind me to do so before doing anything else.
You should at least loosely base this on an expected value calculation of the degree to which the grant increases the probability of achieving the milestones on the list or others like them. You don't need to stick to this formula exactly, and you should feel free to invent new milestones, which you can price by common-sense comparison to the existing ones on the list, and to adjust for subjective and intuitive factors.
You (and the subagent forecasters) may want to consider things like:
The organization's current budget
The organization's past accomplishments (on its past budget)
The organization's ability to scale
The possibility of long-tail outcomes to affect the mean.
The fact that the marginal dollar is rarely more valuable than the average dollar, and so your estimate should remain sensible when multiplied out to the organization's entire budget
The grading criterion is that in the year 2030, a top AI model will try to estimate the utility of the grant. Value the grant as it would look from 2030: count milestones reached by then, plus changes in the probability of later milestones that are already evident by 2030. Discount effects expected after 2030 for the chance that transformative AI has made them moot. State the share of your estimate that comes from post-2030 effects. Please give the true expected value as best you can and do not try to predict a particular evaluator's quirks.
This is a conditional forecast. Please forecast the extra utility from approving this grant, over and above rejecting it. You can assume that, if I rejected it, nobody else would approve it and the organization would have to go without the money. You don't need to worry about the replacement value of money.
Your workflow should be:
Use the skill /grant-adversarial-estimate-v2 (should be available to you) of creating two subagents that each do a forecast separately and then debate it with each other. You (and the subagents) may (and are encouraged to) seek extra information about the organization from the Internet, your training data, and any other source you have access to.
If you run into any theoretical or practical problems with estimating impact, you should be able to find a project document, Autograntor Common Law, with past rulings on these questions. Please follow the instructions in the document for applying past rulings or making your own.
Decide on a mean, median, p10, and p90 outcome for the grant. Remember that the mean may (or may not) be substantially different from the median because of long-tail outcomes, and that some of these numbers (eg p10) may (or may not) be negative if the grant could have side effects.
Report these numbers back to me, along with a table of your expected value calculations - what milestones you used, how much you think the grant would increase the probability of each, and what other things and subjective factors you added where, as well as a summary of your assumptions, uncertainties, and overall thoughts about the process. Please limit your description of the process - what each estimator believed differently, and why you chose one rather than another - to a few sentences, rather than making it an important part of the report.
Report back to me on any problems with the process, which is still new and evolving - I value your input
Depending on which model you are, you may or may not have access to the AI superforecaster FutureSearch. I have generally found it to be less skilled than general frontier models at all tasks EXCEPT explicit and traditional probabilistic forecasts of simple future events. It's also slow and expensive. If you have access, you may consider using it if you run across a traditional forecasting question that seems central to your calculation. Keep in mind that it expects much slower AI progress than the grantmakers you will be assisting, and this may affect its estimates on questions that implicitly rely on AI progress speed; in questions like these, you can side with the grantmakers. Please limit yourself to at most four FutureSearch queries per grant, and report them to me in Step 4.
Thank you. Please let me know if you have any questions.
This should be self-explanatory, except that the /grant-adversarial-estimate-v2 skill tells it a debate procedure it can use: summon Opus and Fable subagents, have them each do the estimate separately, have them each argue for their preferred numbers, and then have the main Claude running the analysis make a final decision. I find that the subagents usually disagree by a factor of 1.5 - 2x, and often do a good job pointing out each other's flaws in a way that helps the main Claude's decisions.
Sample Runs
Here's an example grant forecast, based on Apart Research: Research and Talent Acceleration on Manifund. Because it's an example, I've told it to use only public data. The autograntor found Apart to be relatively cost-effective and good, so I hope they won't mind being analyzed publicly like this. I would be interested in having an Apart representative tell me whether they think this is fair and did a good job surfacing the relevant considerations.
The main takeaway from this is that the grant is valued at 360 points, which suggests a cost-effectiveness of about one point per $2,000, which is good compared to comparable grants. Everything afterward is a discussion of how it arrived at that number.
Apart Research — $800K grant forecast (2026-10-21)
Headline
Size
Mean
Median
P10
P90
P(negative)
Per $1M
Post-2030 share
$800K
360
371
84
625
8%
450
1% strict (CL-18); ≈14% downstream-inclusive
Apart's own average dollar (CL-15, full program vs Apart not existing): ≈750 per $1M. The grant is ≈0.6× that, because without it Apart keeps a slimmer research/evals core plus cheap sprints.
Track record: ≈1,100–1,300 per $1M on ≈$2.4–2.75M spent since 2022.
No values question flips the sign. With branch weights fixed (CL-74), the spread is P10 102, median 372, P90 620. The 16 values corners run from 239 to 396. P(negative) is outcome risk (collapse, bad hires, a management breakdown), not values risk.
Expected-value table (judge's ledger; EV after the execution factor 0.75 and branch weights)
Row
Milestone / price used
With grant → without
EV
Career entries into safety research (≈1.76 extra: 35 extra fellows × 0.04 + ~2.4 extra sprints × 0.15)
Proposed "junior researcher enters sustained AI-safety role" (140); CL-84 for the fellowship, new CL-89 for sprints
P(≥1 entry) 0.83, P(≥2) 0.53, P(≥3) 0.26 (vs 0)
+185
Diffuse upskilling of non-entrants
judgment, vs researcher-year (50)
—
+23
Community / hub network residual
judgment, vs "AI Safety Camp" (4,000) as an upper bound
—
+15
Talent net-negative mirror (values Q1 = no, weight 0.22)
Hostile story on for-profit ties or recruiter fees names the funder (CL-32)
−100
0.05 vs 0.01
−4
Net in-year fundraising effect (CL-34 less CL-4)
—
—
+1
Total
≈360
Execution factor: mean 0.75 with a 7% collapse state, applied to program rows only. Values weights (P yes):
Q1, training many juniors is net positive: 0.78.
Q2, the output is useful rather than noise: 0.75.
Q3, evals carry no capabilities externality: 0.85.
Q4, the EU evals role helps: 0.72.
Correlations: Q1–Q2 0.35, Q2–Q3 0.2.
Assumptions and uncertainties
What the money buys. About 75% (≈$600K) is spent in the plan year: roughly 4 hires after a 2–3 month lag, ~35 extra fellows, ~2–3 extra sprints and ~3 extra peer-reviewed papers. The rest is banked into 2028.
The fellowship. Apart's fellowship is unpaid, part-time (≥10 h/week), remote and PM-mentored, so the strict CL-84 factors give only ≈0.02 career entries per fellow. The judge used 0.04, which is 2× strict, on the grounds that fellows are invited after a selective sprint. Apart's self-reported record (30 placements / 105 fellows) would imply ≈0.08–0.1.
The no-grant world. SFF has had no Apart row since 2024. The last Open Phil grant found was $161K in January 2025, and FutureSearch puts ≥$300K from either funder for 2027 at about 6%. So Apart shrinks rather than closes: probably a 4–6 person core with sprints continuing.
Leadership and team. Kran has been replaced as CEO by Jaime Raldúa Veuthey, and most of the May-2025 team is no longer listed. That is the main reason the execution factor sits at 0.75.
Biggest open fact. Apart's current spend and runway are not public. If its no-grant run-rate is ≥$700K, the mean falls to ≈270. If it is under $450K, the mean rises.
Sensitivities (mean under each change):
Change
Mean
Strict CL-84 rate (0.02 per fellow)
≈300
Applicant-implied rate (0.08 per fellow)
≈480
Estimator A's execution factor
385
Estimator B's execution factor
335
Absorptive share 0.5
≈290
People charge without the refill discount
≈320
Without hire- and funder-specific rows (CL-57)
≈410
How the estimates moved
Mean
Median
P10
P90
Estimator A (Opus)
306
303
84
538
Estimator B (Fable)
420
439
86
709
Judge
360
371
84
625
The estimates passed the agreement test: the means were within 30%, talent was the largest group in both, and no group differed by more than 30% of the larger mean (the talent gap was 89 against a 126 threshold). So the critique round was skipped.
The judge took B's structure for papers and for the career-entry ladder, and A's people charge (at a lower level). The judge cut B's fellow count, since 85 extra fellows a year exceeds anything Apart has run, and cut the paper count to Apart's documented acceptance pace. The judge also corrected A's post-2030 tagging, which double-discounted the 140 milestone under CL-18.
The three decisions that mattered most
Talent.
35 extra fellows × 0.04 per fellow, plus sprint-direct entries at 0.15 per extra sprint, in one Poisson ladder.
The judge verified on Apart's site that the fellowship is unpaid, part-time and remote (apartresearch.com/fellowship).
Papers.
Priced as one Output row plus separate uptake events.
The count is +3 peer-reviewed, not the listing's +5: the 22-paper count was unchanged from May 2025 to March 2026 (Longterm Wiki), and Apart's Nov 2025 EA Forum post shows the 2025 pace.
People charge, −40.
CL-67 applies only to the ~2 of ~4 hires likely to come from other safety orgs.
It does not double-count with the marginal factor: CL-36 puts manager time in the factor, while CL-4 charges the person's output elsewhere.
CL-38 overlaps the banked-program row, so a −15 overlap row is kept.
Facts verified:
apartresearch.com/about: CEO and board.
apartresearch.com/fellowship: terms.
SFF 2024–2026 recommendation pages.
Open Phil grants of Jan 2025 and Dec 2022.
Founders Pledge grantee page (June 2026).
EA Forum, "How Apart Research would use marginal funding" (Nov 2025).
EA Forum, "Where do AI Safety Fellows go?" (600+ alumni dataset).
New Proposed ruling written:claude/common_law_proposed/2026-10-21_apart.md — CL-89 "Career entries caused directly by mass-participation events". It moved the mean by more than 10% (A's sprint row was 78 against the judge's ≈38), and nothing covers it (CL-84 is for mentorships). It is numbered CL-89 because two parallel runs had already used CL-86–88.
Rejected at the bar:
The judge's "execution factor after leadership turnover" ruling. The A-vs-B factor gap moved the mean ±7% around the verdict, and its default range would encode this forecast's parameters.
"Papers are an Output row with use priced separately". This merely refines CL-5's row types plus a CL-30-style audit of claimed counts.
New Proposed milestone: "A peer-reviewed AI-safety paper at a main ML venue from a junior-researcher programme, existence and readership only" | 30 (workshop paper: 8). It is appended to claude/milestones_added.md, and the Doom Debates run's entry is kept.
Values questions (CL-11), priced both ways
Question
Weight (P yes)
Effect of a "no"
Q1: hackathon and fellowship training of juniors is net positive, even though some go to labs or for-profit start-ups
0.78
≈−62
Q2: the output is useful rather than noise or credential dilution
0.75
≈−63 (loses +15, adds −40)
Q3: evals carry no capabilities externality
0.85
≈−22
Q4: the EU AI Office evals role helps
0.72
≈−22
The corners run from 396 (all yes) to 239 (all no).
Open questions for the user or applicant
Apart's actual 2026 spend, cash and runway, and the final 2025 fundraiser total.
The Founders Pledge grant's amount and restriction, and whether any of the six open roles are already funded from outside.
Of the 30 claimed placements: how many were fellows vs sprint-only participants, which organisations, and the fellows' own counterfactual estimate (±25% on the mean).
The date and circumstances of the CEO transition, 2026 paper acceptances, and whether the DarkBench authors are still involved.
FutureSearch
One task with two questions (session ad1a5dec-10f4-458a-b07e-3395e53528b3). It never showed as complete and the results call was refused, so these are the agents' partial conclusions:
Question
Partial answer
Q1: ≥$300K from SFF and/or Open Phil / Coefficient Giving for 2027
≈6%
Q2: Apart still runs sprints plus a fellowship at roughly late-2025 scale in Oct 2027
≈66%
Cost
Calls are tool uses. The Agent results report only total subagent tokens and tool uses; the uncached, cache-read, cache-write and output breakdown is not reported.
Agent
Model
Calls
Tokens
Pre-pass
Sonnet
70 (target ~15)
172,610
Estimator A
Opus
25
144,299
Estimator B
Fable
27
150,147
Critiques
—
not run
—
Judge
Fable
30
178,233
Coordinator
Opus
≈45
not reported
Subagent total
≈645K
Source class: listing-only. Critique round: not run.
Process problems
Pre-pass cost. The pre-pass used ~70 calls against a ~15 target, because the listing was 17 months stale and the current state (CEO, team, budget, funders) had to be pieced together from thin pages.
Suggestion: give stale listings (older than ~12 months) the Commons budget (~30 calls).
Suggestion: add a skill step that asks the plan-year question automatically when the listing is stale.
Concurrent runs. Ruling numbers collided: three runs on the same day each proposed CL-86–88. The staged milestones file also went stale mid-run when another run appended to it. I renumbered and merged by hand.
Suggestion: re-read the proposed folder and milestones file just before writing (the README says this for the folder, but the staging step snapshots early).
Suggestion: use run-scoped provisional numbers that the human user assigns on adoption.
Staging checks. The word-count check can't be done as written: project_read returns the text inline only, so there is no Project-side count to compare against. Staging also means retyping ~25K words of documents, which is the coordinator's largest output cost.
Suggestion: a project_read option that saves to a local file.
Suggestion: a published checksum per document.
simulate.py. Estimator A tagged the 140-milestone talent rows with post2030 = 0.25, which CL-18 forbids because the milestone is already a 2030-judged price. Suggestion: a simulate.py warning when a row priced on a generic or milestone unit carries post2030 > 0.
The sign of the estimate is robust. The level hinges on two facts only the applicant can supply: the current run-rate, and the counterfactual career-entry rate per fellow. One email to Apart would probably narrow the range more than another estimator would.
Some things I want to highlight from this report:
It's much more complicated than my description of the autograntor process suggests. This is because the AIs decided to do a lot of this (eg the sensitivity analysis, the "execution factor", etc) themselves, based on their impression of what a good forecast looked like, and I let them. I'm left in the position of trying to catch up to and understand their work.
It made two FutureSearch calls - one to determine whether anyone else would give Apart a grant, and another to determine with Apart would still run their sprints and fellowship in 2027. In this run, the FutureSearch calls failed, which is a constant hazard of workflows requiring different AIs to connect to each other, and probably was because I failed to top up my tokens properly. I can't tell whether the probability it ultimately got was its own guess (Claude is pretty good at this, although still worse than FutureSearch) or its reading some intermediate result off the FutureSearch connector.
It thinks in broad strokes: the output of Apart is "alignment papers" and "alignment researchers", which are assumed to be about as good as the average member of their category. It's capable of thinking more qualitatively than this - you can see it price in a 10% increased chance that the grant produces a "standout" result of the same importance as "DarkBench" (I didn't personally know what this was, but I looked it up, and it's a previous Apart result that got lots of traction - this is a good example of how the AIs know more than I do in a productive way). But I don't really get the impression that if Apart were actually much more on the ball than other alignment organizations and their unique paradigm was about to solve alignment forever - or if they were much less on the ball than other alignment organizations, and a complete sham pumping out reasonable-sounding but ultimately slop papers - the autograntor would be capable of noticing this or accounting for it (then again, I personally also might not be able to do this).
It cost somewhere north of 600,000 tokens. This is lower than other runs - this might have been one of my abbreviated-process test runs, I can't remember. An average autograntor run on Opus 5.5 High Effort uses about 30% of my five-hour limit (I'm on Claude Max plan). I think this works out to about $20 in API credits. It usually takes 20 - 60 minutes.
Doom Debates — $200K grant forecast (independent re-run, 2026-10-22)
Common Law 2026-10-20.1 + proposed through 2026-10-21_separatrix.md as staged (CL-84, 85, 88). At the user's request this run deliberately excluded 2026-10-21_doomdebates.md and the Proposed milestone from the earlier Doom Debates run, and no agent read the earlier report until the verdict was final. Workflow: grant-adversarial-estimate-v2; listing-only source (Manifund page + public web); critique round ran.
Headline (judge)
Size
Mean
Median
P10
P90
P(neg)
Per $1M
Post-2030 share
$200K
72
60
−7
154
12% (10% with values fixed)
359
9% strict (CL-18); ≈15% incl. downstream
Organisation's average dollar (CL-15): ≈1,230 per $1M on cash, ≈615 with the host's unpaid time imputed. The grant is worth about 0.3 of the average cash dollar because the host keeps the show going without it; the grant buys roughly a third more 2027 output (producer, interns, marketing).
Values corners (CL-74): 54 (all three values questions go against) to 82 (all go for). No corner is negative; P(negative) is outcome risk (collapse, hostile-story worlds, talent row not firing).
Most decision-relevant sensitivity: no-grant 2027 fundraising. At $40K the mean is ≈95; at $120K ≈50 (central ≈$85K including a 20% chance the pending FLI media application pays).
EV table (judge ledger, after branch weights and execution factor mean 0.85, 4% collapse)
Row
Type / milestone
Basis
Grant ΔP or PERCYs
Value
Talent: ≥1 viewer enters sustained safety/advocacy work because of the grant
CL-82 reversal of 0.4 of glance/long-form rows; CL-32 doubling
−3.7
Values: attacks on movement figures (P 0.45 harm)
small mirror rows
−1.3
Total
71.8
Assumptions and uncertainties
No-grant world: the show continues at lower volume and quality with the unpaid host (his stated plan); all donations already received stay in both worlds (CL-8, CL-43, CL-88); this $200K is not replaced. About 85% is 2027 spend, 15% banked.
Audience: 149K subscribers, flat; ~7K median views per recent episode; all on-topic (CL-30); 80% US/UK/Canada (host's Feb 2026 talk) → CL-71 factor 0.65.
Softest inputs: the elite headcount (250 lab/government staff, guessed by both estimators; at Rob Miles's Δ_E the mean rises ~22, at 100 staff it falls ~13) and the talent rate (no documented viewer has entered safety work because of the show).
I priced none of the applicant's 10–100× growth ambition beyond the breakout row; recent growth is flat.
How the estimates moved
A (Opus) 68, B (Fable) 127; talent and continuation groups diverged, so the critique round ran. The judge sided with A on the units and caps (talent rate applies to new core viewers, CL-25 cap 500, continuation base excludes by-2030 rows) and with B on direction (no explicitness discount for a show that names the field; breakout window runs to 2030), landing at 72.
Comparison with the earlier run (read only after this verdict)
The earlier run gave mean 67, median 49, P10 −15, P90 169. This re-run: 72, 60, −7, 154. The estimator split recurred almost exactly (46/120 then, 68/127 now), and both judges decided the same three disputes (talent rate, elite headcount, breakout) in the same direction. The judge here independently re-derived a "thinner program" marginal-factor rule that the earlier run had already proposed as CL-87, so I did not propose it again.
New Proposed ruling (file claude/common_law_proposed/2026-10-22_doomdebates-rerun.md): CL-90, CL-26's rate for ongoing channels (apply to new core joiners, not the whole core yearly; no explicitness cut when the work names the field; price entrants by research/advocacy mix). The earlier run rejected a narrower version at the bar as a refinement of CL-26; two independent runs hitting the same issue suggests it deserves a ruling or a CL-26 amendment.
Dropped at the bar: "marginal factor for a grant that thickens a program that continues without it" — covered by CL-87 (2026-10-21_doomdebates.md). Refinement notes: CL-38's "plan-year increment" should mean annual-flow rows only; CL-25's media step row uses the by-2030 window and one owned-channel threshold.
New Proposed milestone: "a viewer enters sustained AI-safety advocacy or communications work by 2030 because of a media grant", 55.
No disagreements with Adopted rulings.
FutureSearch (one session, two questions; session 00af7fba-7c44-4736-b2ed-489c7a6b180a)
Neither returned a final results table; figures are the agents' own conclusion lines from the progress log.
≥350K YouTube subscribers on 31 Dec 2027: ≈4% (threshold set too high to be very informative).
≥36 episodes published in 2027: ≈87%.
Open questions
Cash on 1 Jan 2027, the 2027 donor pipeline, and whether the FLI media application has resolved (moves the mean ≈50–95).
Without this money, does the producer stay part-time, and at what pay?
Any outcome data (PauseAI referrals, viewer surveys, named viewers who entered safety or advocacy work)?
Audience composition: how many lab or government AI staff follow monthly?
Cost
Agent
Model
Tool uses
Tokens (total)
Coordinator
Opus
~45
not reported
Pre-pass
Sonnet
56 (target ~15)
180K
Estimator A + critique
Opus
32 + 6
188K cumulative
Estimator B + critique
Fable
28 + 6
181K cumulative
Judge
Fable
37
197K
The split into uncached, cached and output tokens is not available from the tool results. Source class: listing-only. Critique round ran.
Process notes
Ruling numbers are colliding: CL-86 and CL-87 each exist twice (2026-10-21_doomdebates.md and 2026-10-22_forethought.md), and CL-89 was taken by the Apart run. Runs in parallel sessions each pick "the next free number". A per-run prefix (e.g. CL-2026-10-22-a) or a numbering check at write time would fix this.
Concurrent runs also write milestones_added.md; this run found an Apart entry added mid-run and merged it before uploading. A whole-file overwrite can silently drop another run's entry.
The pre-pass took 56 tool calls against a target of 15, mostly scraping Manifund and YouTube HTML. Attaching the listing text would cut this.
Staging by re-typing project_read text is the most expensive coordinator step; the word-count check compares the copy with itself, so it verifies nothing.
The critique round again moved nothing directly (each critic recomputed the other near itself); the judge did the real work.
This was part of an experiment testing "high" vs. "medium" effort autograntor workflows. The results were relatively encouraging - a previous medium-effort analysis got a mean of 67 units; this high-effort replication got a mean of 72. My weak conclusion is that there's not much benefit to high effort, though I want to replicate this a few more times to be sure.
So . . . what now?
Is the autograntor good?
The autograntor generally agreed with the gestalt consensus of Lightcone Commons evaluators. I say "gestalt consensus" because each evaluator made their own grant list, and each was very different. But for example, the autograntor's top choice was an organization that I and two other human evaluators also put as their top choice, and several other human evaluators ranked highly. I worried that it might be cheating by checking other people's beliefs, so I re-ran it without letting it use the connection, and it gave a similar answer, although I can't say for sure that it didn't find some way around my restrictions.
I also ran it on some of the worst grants from Lightcone Commons, a spam application by an unknown-to-us person that no evaluator found any value in. It ranked it about 100x less cost-effective than the real grants.
I also ran versions of the autograntor on four different AIs, to see if they agreed with one another. This didn't really work. Two of them - FutureSearch and Preseen - were superforecasting-specialized AI that became very confused by the whole thing, occasionally lost the attachments and hallucinated their contents, and produced results off by an order of magnitude. Of the remaining two - Claude and ChatGPT - ChatGPT consistently gave results 2-3x better (more cost-effective) than Claude for everything. Every time I showed the AIs one another's results, Claude stuck to its guns and insisted that it's right, and ChatGPT admitted that it's wrong and deferred to Claude. I appreciate both AIs for their honesty and consistency, but this makes it hard to judge inter-rater reliability. The correlation between Claude and ChatGPT's results is 0.92, but again, this doesn't mean much given the orders-of-magnitude differences involved.
What about my subjective impression?
The autograntor is harder-working than I am. It does more research, surfaces more considerations, and catches things that I miss (I can say this with certainty, because some other Lightcone Commons evaluators are also better researchers than I am, and the autograntor often found the same things they did, even when I tried to prevent it from directly copying their work).
The autograntor is better at modeling than I am. It thinks in terms of distributions more often than I do, is more likely to test its assumptions with sensitivity analyses, and has more principled and consistent beliefs about the mathematics of credit assignment.
At this point, the autograntor never gives insane results when run on Claude (it still does on the superforecasters, and to a lesser degree on Astra). I'm nervous about this, because it used to give some insane results, and I sort of vibe-coded fixes to them, and I worry that I spotted the obvious ones that ruined the results according to my intuitions, but that there are many smaller ones that usually cancel out or are too small to notice.
(exception: it sometimes makes new insane common law rulings, and these silently poison several runs before they poison one loudly enough that I notice, investigate, and correct it. The time I'm thinking of, I can see in retrospect what it was thinking, and it would have been a reasonable way to run an autograntor, but it wasn't how I was interpreting its results and it would have misled me if I had interpreted them in the way that seemed natural to me)
The autograntor is sufficiently "more sophisticated" than I am (more prone to complicated analysis) that I don't feel capable of directly judging its reasoning, instead having to resort to indirect methods like comparing different AIs' output and asking them to criticize each other. I believe many parts of it are still subtly flawed. The main question is whether these subtle flaws make using it worse than the job I would do unassisted. I plan to test the autograntor in my next Lightcone Commons grant round, hopefully before any of the other evaluators have formed opinions, then check how well it matches their outcomes.
I don't know whether using the autograntor is an implicit decision to ignore subtle non-quantitative considerations. For example, in Lightcone grantmaking meetings, we often debated how to think about productive and successful organizations run by potentially immoral people. Some grantmakers wanted to support the organizations so they could continue their productive and successful work; others feared they would be further empowering the immoral people and letting them corrupt the community. The autograntor seems aware of these considerations, and often includes a negative term for downside risk in its analyses - things like "promotes false ideas" or "causes a scandal". But it can only consider the most quantifiable versions of these harms, not some kind of subtle year-by-year chipping away at norms that might one day snowball into a disaster.
I would like to keep using my own reasoning, with the autograntor as a supplement, but this has been increasingly hard. The lure of a specific cost-effectiveness number attached to each grant is a powerful anchor, even though I know how many tries it took before the autograntor numbers looked sane, and how many flaws it probably still has. I would like to be able to double-check each row in its analysis, substitute my own guesses, and then adjust for some sort of gestalt subjective factor that mere computers could never consider. But in fact, I find myself glazing over the analysis as I read it, thinking "yeah, that number is probably right . . . yeah, that number is probably right too".
(for example, did you catch that in the Doom Debates analysis above, the "PERCYs E-class" row basis calculation outputs 45,900 PERCYs, but the table lists 21,600 PERCYs instead? I didn't until a few minutes before posting this on Less Wrong, when I asked an AI to proofread this essay for my typographical errors and it found that error too. I challenged the Claude instance that did the Doom Debates analysis, and it said that the 0.68 in "basis" was the retained share without the grant, so the value of the grant was 1 - 0.68 in that term, and it just wrote the calculation in a very confusing way, which is fine, except that if I had been reading closely I would have noticed that discrepancy earlier).
Instructions for replicating/testing this
Ideally, other people would look at this project, test its failure modes, and investigate its potential. I'm posting it here to start this process.
Give the skill grant-adversarial-estimate to Claude and ask it to install it in itself. Then start a new session with Opus 5.5, medium effort, and prompt it with:
Prompt
I would like your help with AI safety grantmaking, using a specific grant forecasting system. I've attached four related documents:
a list of utility weights for various milestones
a description of PERCYs, a separate module of the model used for persuasion/communication grants. You don't have to consult this if you don't think it's relevant. If you do use it, use the PERCYs-to-utility conversion on the main milestones doc.
The "Common Law" - a set of rulings that past AIs have made to resolve edge cases
the grant proposal and any peripheral documents related to the grant.
The grant is [NAME OF GRANT]
Please forecast how much extra utility (based on the milestones doc) the grant will provide according to the grading criterion below, at the following amounts:
[AMOUNTS]
If I forgot to provide any of the documents, or to fill in the grant amounts, please remind me to do so before doing anything else.
You should at least loosely base this on an expected value calculation of the degree to which the grant increases the probability of achieving the milestones on the list or others like them. You don't need to stick to this formula exactly, and you should feel free to invent new milestones, which you can price by common-sense comparison to the existing ones on the list, and to adjust for subjective and intuitive factors.
You may want to consider things like:
The organization's current budget
The organization's past accomplishments (on its past budget)
The organization's ability to scale
The possibility of long-tail outcomes to affect the mean.
The grading criterion is that in the year 2030, a top AI model will try to estimate the utility of the grant. Value the grant as it would look from 2030: count milestones reached by then, plus changes in the probability of later milestones that are already evident by 2030. Discount effects expected after 2030 for the chance that transformative AI has made them moot. State the share of your estimate that comes from post-2030 effects. Please give the true expected value as best you can and do not try to predict a particular evaluator's quirks.
This is a conditional forecast. Please forecast the extra utility from approving this grant, over and above rejecting it. You can assume that, if I rejected it, nobody else would approve it and the organization would have to go without the money. You don't need to worry about the replacement value of money.
Your workflow should be:
1. Use the skill /grant-adversarial-estimate-v2 (should be available to you) of creating two subagents that each do a forecast separately and then debate it with each other. You (and the subagents) may (and are encouraged to) seek extra information about the organization from the Internet, your training data, and any other source you have access to.
2. If you run into any theoretical or practical problems with estimating impact, you should be able to find a project document, Autograntor Common Law, with past rulings on these questions. Please follow the instructions in the document for applying past rulings. You're not currently in communication with the main project, so there's no point in adding your own.
3. Decide on a mean, median, p10, and p90 outcome for the grant. Remember that the mean may (or may not) be substantially different from the median because of long-tail outcomes, and that some of these numbers (eg p10) may (or may not) be negative if the grant could have side effects.
4. Report these numbers back to me, along with a table of your expected value calculations - what milestones you used, how much you think the grant would increase the probability of each, and what other things and subjective factors you added where, as well as a summary of your assumptions, uncertainties, and overall thoughts about the process.
...and attach these three files, plus the grant proposal:
Milestones (on my worldview; warning that if you use this file without modification you're implicitly accepting all of my controversial beliefs)
This setup assumes you won't have a FutureSearch account, and so it's missing the superforecasting calls. If you want to add that in, get an account at FutureSearch.ai, connect it using the guide here, and add the following paragraph to your prompt.
FutureSearch prompt addition
You have access to the AI superforecaster FutureSearch. I have generally found it to be less skilled than general frontier models at all tasks except explicit and traditional probabilistic forecasts of simple future events. It's also slow and expensive. If you have access, you may consider using it if you run across a traditional forecasting question that seems central to your calculation. Keep in mind that it expects much slower AI progress than the grantmakers you will be assisting, and this may affect its estimates on questions that implicitly rely on AI progress speed; in questions like these, you can side with the grantmakers. Please limit yourself to at most four FutureSearch queries per grant, and report them to me in Step 3.
The best project I tested had a cost-effectiveness of one point per $1-$2K. Excellent projects that most grantmakers were happy to fund had cost-effectivenesses of 1 / $2-$5K. Decent projects that some grantmakers funded and others didn't had cost-effectivenesses of $5-$10K, and projects that we took seriously but mostly didn't end up funding unless there were hard-to-quantify reasons in favor were usually in the range of 1 / $10-$20K. Beyond $20K was the line where the human grantmakers I observed generally didn't treat the projects as serious contenders.
Feel free to send me any reports you get, especially if you find any unusual behavior or insane results. The autograntor isn't currently set up to evaluate any grants beyond standard AI-safety-related grants, and I don't expect it to perform well in any other field.
Why didn't I assign saving the world a value of one million points, implicitly pricing everything else in microdooms? I felt less certain that I could have an intuitive sense of how many microdooms the average alignment paper averts than that I could intuitively rank good grant outcomes relative to each other. Since on my model Plan A would decrease p(doom) by 20%pp, conversion to a microdoom based framework is trivial (within the model - if someone else has a different starting p(doom), they have to convert differently)
If I'd been smart, I would have filled them in personally first, without AI help. But if I did this, I've since lost the file, and now I'm contaminated by knowing the AIs' values.
I also checked this by having an AI design itself a tournament where it showed itself two milestones at a time, rated how much better one was than another, and used an Elo-like algorithm to turn all of this into a set of consistent weights. This didn't change things much.
I think the rating of one point is indefensible on community-building grounds, but the whole point of this exercise is to quantify worldviews that differ from mine.
Again, something must be wrong with these numbers - a really great alignment paper is worth 5,000, so this suggests that the great alignment paper = convincing five million people, whereas common-sensically, convincing five million people is clearly better. But convincing every non-elite person in the US (let's say 10% of people are elites, so this is about 300 million), would be worth 300,000 points, or about a third of Plan A, which actually does seem order-of-magnitude right (if everyone were convinced, this would be a huge, but not completely sufficient, step to getting Plan A). My guess is either that this has something to do with double-counting, or that I'm failing to model elites-vs-normies correctly - it could be that on an individual level, elites are worth vastly more than normies, but convincing all normies at once means elites are likely to go along (or is simply nonsensical, since it's hard to imagine all normies but no elites being convinced).
Many of these grants are on Lightcone Commons. Thank you to Habryka and the Lightcone team for creating an AI connector that allows the autograntor to access the Lightcone Commons site directly. When a grant is on LC, the autograntor is able to read the application itself, the other evaluators' notes, and the transcripts of interviews with the grantees, which it takes seriously and adds to its estimates. I don't include any Lightcone-based analyses in the sample analysis section below, because I don't know what the privacy norms are and don't want to leak information from the applications. There's some risk that this access "contaminates" the autograntor, since it can see what humans (including me) thought. In my brief tests, this hasn't shifted its judgment noticeably.
“I would like to be able to double-check each row in its analysis… But in fact, I find myself glazing over the analysis as I read it, thinking "yeah, that number is probably right . . . yeah, that number is probably right too".
Reading this made me wonder whether organizations, as they hand off increasing parts of their judgment to AI, should have an employee randomly audit some small sample of the AI’s conclusions without using so much as GPT-3. This might be especially helpful for domains like grantmaking, where it’s high stakes and the feedback loops are long and noisy.
I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathematical models of the outcome distribution, or of where the marginal dollar stops being highly effective.
AI has been a godsend for all of this. It can research, model, and - as of this year - superforecast outcomes. It can do what would be hours of work for a human in minutes. I found myself using it more and more. But however much I used it, it always felt like I was the bottleneck. The parts I could outsource to AI were sparkling, perfect, exhaustive - but they were all tacked together with shoddy connective tissue formed by the Mark 1.0 human brain. So I wondered if I could automate grantmaking.
I started with the dumbest possible strategy. I assigned the best thing I could think of - a well-crafted US-China pause based on the AI Futures Project's Plan A - an arbitrary value of one million points.[1] Then I generated about eighty lesser AI safety milestones - from MATS to HPMOR to the RAISE Act to the emergent misalignment paper - and subjectively assigned them some number of points based on how good they seemed relative to Plan A and each other. Then I fed it to the AI to give it a sense for how good a "point" was in this framework. Then I asked it to forecast how many points various grant proposals were worth. The system has since accrued lots of extra epicycles and features and complications, but that's still the skeleton.
As far as I can tell, this dumbest possible strategy worked well. The autograntor gives common-sensical results that match my impressions and the vague gestalt funding order of Lightcone grantors. Its reports surface all relevant information, and its reasoning is clear. The single highest cost-effectiveness grant it's found so far is also the one that got the most interest during the last Lightcone run.[2] When I tested it on a bottom-tier proposal that no human evaluator even considered, it gave cost-effectiveness numbers two orders of magnitude worse than the genuinely-under-consideration grants that I had tested it on before.
This post will present details for people who want to examine these ideas or build on them further. The current version of the autograntor has four parts: a Milestones document, a FutureSearch connector, a PERCYs module, and a Common Law. I'll start by reviewing each, then give some sample grant analyses, then attach the files you need to run the autograntor yourself.
The Milestones
The milestones are the earliest place that this strategy could fail. Can I really translate my beliefs about what matters in AI safety into a list of milestones, each with a consistent point value?
This combines two questions. First, can I, Scott, generate a set of milestone point values that accurately and consistently conveys my own worldview? Second, if someone else were to translate their own worldview into milestones, would it agree with mine?
I chose to deprioritize the second question. Yes, I have different beliefs about AI safety than other knowledgeable people, but this is an inherent problem for my grantmaking, even if I did it manually. If we could solve the first problem, then I could be confident that the autograntor was correctly representing my opinions, and someone else who wanted to use an autograntor could confidently ask it to represent their opinions instead.
To investigate the first question, I asked four AIs - Opus, Fable, ChatGPT, and Grok - to guess how I, Scott, would fill them in.[3] If all four got similar results, then "Scott's worldview" constrains the point values well; if all of our results were very different, then noise in the translation of a worldview into numbers overwhelms signal.
Two-way correlations ranged from 0.95 - 0.99, which might sound extreme, but some of the milestones were obviously many orders of magnitude better than other milestones - it's not hard to tell that achieving global pause with China is much better than successfully running the Manifest conference.
A better measure is the geometric-mean difference between two AIs across all milestones. This ranged from 1.6x (Opus vs. ChatGPT) to 2.6x (ChatGPT vs. Grok).
Sample table of AI milestone predictions
Since my true milestone value list came from taking the average of many different methods of calculation, the real variance is something like the square root of these numbers, ie from 1.3x to 1.6x. I consider this good enough to work with.[4]
I would have liked to have gotten some humans to try to predict my point value within my world-view, but couldn't get anyone to agree to do this (except for one person who agreed but used an AI, unintentionally duplicating my procedure).
Although I don't think the second question (difference between people with different worldviews) matters for the success of the autograntor, I did investigate it via a survey where I asked people to fill in their own values. Nine people agreed, including four people whose names I recognized and who I expected to have good models (other people might also have good models, I just didn't recognize their names). The geometric mean difference between myself and those four people was 2.9x, 4.4x, 6.9x, and 8x, so much larger than the difference between different AIs trying to estimate my view. Outside that artificial limitation, geometric means were even higher - for example, one pair had a geomean difference of 15x! To give an example of where such differences came from, "Harry Potter And The Methods Of Rationality" was an especially controversial milestone - one person rated it as 1 point, another person as 75,000 points.[5]
Everything below assumes that the Milestones list is good enough to work with.
The FutureSearch Connector
FutureSearch is an AI superforecaster. In last season's Metaculus Cup, it took third place among bots, and separately would have taken third place among humans if it had been human. I think these differences are mostly noise, and it should be considered approximately as good as the top human superforecasters. I wanted to take advantage of this skill to predict grant outcomes.
My original plan was to run the autograntor entirely on FutureSearch. This failed badly. FutureSearch wasn't designed to do things more complex than a simple clearly-framed forecast, so it bungled the various calculations involved in the autograntor process.
What seems to have worked is to run the autograntor on Claude Opus, but give it a connector that allows it to call FutureSearch via the API for all components of the grant analysis that benefit from a probabilistic forecast. For example, in one case where the success of an organization depended on it getting a new CEO quickly, Claude gave the relevant information to FutureSearch and asked it to predict the probability that it would find a new CEO within six months. In another case where an organization was working on verification technology, Claude asked FutureSearch to predict whether the political situation would lead to a treaty where the relevant verification was necessary.
FutureSearch has kindly given me a free subscription, but if you're paying through the API, it's about $1 - $5 per query like this. I asked Claude to limit itself to five queries per grant analysis.
Preseen and Mantic have competing AI superforecaster products with equal evidence of accuracy. I'm using FutureSearch because I'm most familiar with it, it seems to be the fastest (unless there are ways to speed up the others that I don't know about), and it seems to have the most convenient connector to Claude (unless the others have connectors I haven't noticed yet). Future autograntor models might explore the advantages of these other products, or try to aggregate forecasts from all of them.
PERCYs
Many grants' theory of change focused on persuasion. There were a wide variety of persuasion grant types, from PLZDONTKILLUS (ordinary people watch short videos and become more informed/concerned about AI) to Palisade (demos to Congressmen make them more informed/concerned about AI).
Rather than force this into the milestone system, I created a unit of persuasion, the PERCY (Persuasion-Year), equal to flipping one average person from neutral about AI risk to concerned about AI risk, for one year. Through a process similar to the Milestones, I then tried to assign multipliers to various factors - what if the person is a journalist? a Congressman? a frontier company employee? Xi Jinping? what if instead of just making them concerned, it turns them into a dedicated activist?
This system started as another attempt to convert AI superforecasters' predictive ability into grantmaking ability. The superforecaster can predict that a certain channel we're considering a grant to will get 500,000 viewers, of which 3 will be Congressional staffers and 6 will work at frontier labs, and then I can turn that prediction into a step in a grant evaluation.
There are currently some serious flaws in this scheme - most notably, every method of adjusting for double-counting (the fact that two properties might be trying to persuade the same audience) fails in a different way. I'm currently banking on these absolute failures still allowing for relative comparison - ie the results when comparing different grants are still somewhat sensible. As a stopgap, in some of the documentation I've replaced the absolute measure of PERCYs (which is a probably-false claim about the absolute number of people persuaded) with a relative measure, "miles per hour" (a comparison to the number of people persuaded by one hour of Rob Miles video content).
One of the Milestones on my list is the value of a PERCY. I currently rank 1000 PERCYs = about 1 point.[6]
The Autograntor Common Law
Autograntor analyses often produce difficult questions of interpretation. For example:
At first, I despaired of ever solving all of these. Later...
...I decided to hand them off to the AI as they came up. I told the AI to make its best judgment according to the spirit of the exercise and record its ruling in a Common Law document. Later autograntor runs would respect previous rulings for the big questions and add their own sub-rulings for smaller sub-questions. Every so often, I would review the Common Law and make sure everything in it was sensible and matched the spirit of the exercise. This ensures that at the very least, all grants will be judged according to similar criteria.
The current Common Law is up to about a hundred rulings. Predictably, I'm delinquent in my resolution to review it regularly.
The Prompt
Here's how I start the autograntor:
This should be self-explanatory, except that the /grant-adversarial-estimate-v2 skill tells it a debate procedure it can use: summon Opus and Fable subagents, have them each do the estimate separately, have them each argue for their preferred numbers, and then have the main Claude running the analysis make a final decision. I find that the subagents usually disagree by a factor of 1.5 - 2x, and often do a good job pointing out each other's flaws in a way that helps the main Claude's decisions.
Sample Runs
Here's an example grant forecast, based on Apart Research: Research and Talent Acceleration on Manifund. Because it's an example, I've told it to use only public data. The autograntor found Apart to be relatively cost-effective and good, so I hope they won't mind being analyzed publicly like this. I would be interested in having an Apart representative tell me whether they think this is fair and did a good job surfacing the relevant considerations.
The main takeaway from this is that the grant is valued at 360 points, which suggests a cost-effectiveness of about one point per $2,000, which is good compared to comparable grants. Everything afterward is a discussion of how it arrived at that number.
Some things I want to highlight from this report:
It's much more complicated than my description of the autograntor process suggests. This is because the AIs decided to do a lot of this (eg the sensitivity analysis, the "execution factor", etc) themselves, based on their impression of what a good forecast looked like, and I let them. I'm left in the position of trying to catch up to and understand their work.
It made two FutureSearch calls - one to determine whether anyone else would give Apart a grant, and another to determine with Apart would still run their sprints and fellowship in 2027. In this run, the FutureSearch calls failed, which is a constant hazard of workflows requiring different AIs to connect to each other, and probably was because I failed to top up my tokens properly. I can't tell whether the probability it ultimately got was its own guess (Claude is pretty good at this, although still worse than FutureSearch) or its reading some intermediate result off the FutureSearch connector.
It thinks in broad strokes: the output of Apart is "alignment papers" and "alignment researchers", which are assumed to be about as good as the average member of their category. It's capable of thinking more qualitatively than this - you can see it price in a 10% increased chance that the grant produces a "standout" result of the same importance as "DarkBench" (I didn't personally know what this was, but I looked it up, and it's a previous Apart result that got lots of traction - this is a good example of how the AIs know more than I do in a productive way). But I don't really get the impression that if Apart were actually much more on the ball than other alignment organizations and their unique paradigm was about to solve alignment forever - or if they were much less on the ball than other alignment organizations, and a complete sham pumping out reasonable-sounding but ultimately slop papers - the autograntor would be capable of noticing this or accounting for it (then again, I personally also might not be able to do this).
It cost somewhere north of 600,000 tokens. This is lower than other runs - this might have been one of my abbreviated-process test runs, I can't remember. An average autograntor run on Opus 5.5 High Effort uses about 30% of my five-hour limit (I'm on Claude Max plan). I think this works out to about $20 in API credits. It usually takes 20 - 60 minutes.
Here's a second example, based on Liron Shapira's Doom Debates' Manifund application here.
This was part of an experiment testing "high" vs. "medium" effort autograntor workflows. The results were relatively encouraging - a previous medium-effort analysis got a mean of 67 units; this high-effort replication got a mean of 72. My weak conclusion is that there's not much benefit to high effort, though I want to replicate this a few more times to be sure.
So . . . what now?
Is the autograntor good?
The autograntor generally agreed with the gestalt consensus of Lightcone Commons evaluators. I say "gestalt consensus" because each evaluator made their own grant list, and each was very different. But for example, the autograntor's top choice was an organization that I and two other human evaluators also put as their top choice, and several other human evaluators ranked highly. I worried that it might be cheating by checking other people's beliefs, so I re-ran it without letting it use the connection, and it gave a similar answer, although I can't say for sure that it didn't find some way around my restrictions.
I also ran it on some of the worst grants from Lightcone Commons, a spam application by an unknown-to-us person that no evaluator found any value in. It ranked it about 100x less cost-effective than the real grants.
I also ran versions of the autograntor on four different AIs, to see if they agreed with one another. This didn't really work. Two of them - FutureSearch and Preseen - were superforecasting-specialized AI that became very confused by the whole thing, occasionally lost the attachments and hallucinated their contents, and produced results off by an order of magnitude. Of the remaining two - Claude and ChatGPT - ChatGPT consistently gave results 2-3x better (more cost-effective) than Claude for everything. Every time I showed the AIs one another's results, Claude stuck to its guns and insisted that it's right, and ChatGPT admitted that it's wrong and deferred to Claude. I appreciate both AIs for their honesty and consistency, but this makes it hard to judge inter-rater reliability. The correlation between Claude and ChatGPT's results is 0.92, but again, this doesn't mean much given the orders-of-magnitude differences involved.
What about my subjective impression?
The autograntor is harder-working than I am. It does more research, surfaces more considerations, and catches things that I miss (I can say this with certainty, because some other Lightcone Commons evaluators are also better researchers than I am, and the autograntor often found the same things they did, even when I tried to prevent it from directly copying their work).
The autograntor is better at modeling than I am. It thinks in terms of distributions more often than I do, is more likely to test its assumptions with sensitivity analyses, and has more principled and consistent beliefs about the mathematics of credit assignment.
At this point, the autograntor never gives insane results when run on Claude (it still does on the superforecasters, and to a lesser degree on Astra). I'm nervous about this, because it used to give some insane results, and I sort of vibe-coded fixes to them, and I worry that I spotted the obvious ones that ruined the results according to my intuitions, but that there are many smaller ones that usually cancel out or are too small to notice.
(exception: it sometimes makes new insane common law rulings, and these silently poison several runs before they poison one loudly enough that I notice, investigate, and correct it. The time I'm thinking of, I can see in retrospect what it was thinking, and it would have been a reasonable way to run an autograntor, but it wasn't how I was interpreting its results and it would have misled me if I had interpreted them in the way that seemed natural to me)
The autograntor is sufficiently "more sophisticated" than I am (more prone to complicated analysis) that I don't feel capable of directly judging its reasoning, instead having to resort to indirect methods like comparing different AIs' output and asking them to criticize each other. I believe many parts of it are still subtly flawed. The main question is whether these subtle flaws make using it worse than the job I would do unassisted. I plan to test the autograntor in my next Lightcone Commons grant round, hopefully before any of the other evaluators have formed opinions, then check how well it matches their outcomes.
I don't know whether using the autograntor is an implicit decision to ignore subtle non-quantitative considerations. For example, in Lightcone grantmaking meetings, we often debated how to think about productive and successful organizations run by potentially immoral people. Some grantmakers wanted to support the organizations so they could continue their productive and successful work; others feared they would be further empowering the immoral people and letting them corrupt the community. The autograntor seems aware of these considerations, and often includes a negative term for downside risk in its analyses - things like "promotes false ideas" or "causes a scandal". But it can only consider the most quantifiable versions of these harms, not some kind of subtle year-by-year chipping away at norms that might one day snowball into a disaster.
I would like to keep using my own reasoning, with the autograntor as a supplement, but this has been increasingly hard. The lure of a specific cost-effectiveness number attached to each grant is a powerful anchor, even though I know how many tries it took before the autograntor numbers looked sane, and how many flaws it probably still has. I would like to be able to double-check each row in its analysis, substitute my own guesses, and then adjust for some sort of gestalt subjective factor that mere computers could never consider. But in fact, I find myself glazing over the analysis as I read it, thinking "yeah, that number is probably right . . . yeah, that number is probably right too".
(for example, did you catch that in the Doom Debates analysis above, the "PERCYs E-class" row basis calculation outputs 45,900 PERCYs, but the table lists 21,600 PERCYs instead? I didn't until a few minutes before posting this on Less Wrong, when I asked an AI to proofread this essay for my typographical errors and it found that error too. I challenged the Claude instance that did the Doom Debates analysis, and it said that the 0.68 in "basis" was the retained share without the grant, so the value of the grant was 1 - 0.68 in that term, and it just wrote the calculation in a very confusing way, which is fine, except that if I had been reading closely I would have noticed that discrepancy earlier).
Instructions for replicating/testing this
Ideally, other people would look at this project, test its failure modes, and investigate its potential. I'm posting it here to start this process.
Give the skill grant-adversarial-estimate to Claude and ask it to install it in itself. Then start a new session with Opus 5.5, medium effort, and prompt it with:
Prompt
...and attach these three files, plus the grant proposal:
This setup assumes you won't have a FutureSearch account, and so it's missing the superforecasting calls. If you want to add that in, get an account at FutureSearch.ai, connect it using the guide here, and add the following paragraph to your prompt.
FutureSearch prompt addition
You have access to the AI superforecaster FutureSearch. I have generally found it to be less skilled than general frontier models at all tasks except explicit and traditional probabilistic forecasts of simple future events. It's also slow and expensive. If you have access, you may consider using it if you run across a traditional forecasting question that seems central to your calculation. Keep in mind that it expects much slower AI progress than the grantmakers you will be assisting, and this may affect its estimates on questions that implicitly rely on AI progress speed; in questions like these, you can side with the grantmakers. Please limit yourself to at most four FutureSearch queries per grant, and report them to me in Step 3.
The best project I tested had a cost-effectiveness of one point per $1-$2K. Excellent projects that most grantmakers were happy to fund had cost-effectivenesses of 1 / $2-$5K. Decent projects that some grantmakers funded and others didn't had cost-effectivenesses of $5-$10K, and projects that we took seriously but mostly didn't end up funding unless there were hard-to-quantify reasons in favor were usually in the range of 1 / $10-$20K. Beyond $20K was the line where the human grantmakers I observed generally didn't treat the projects as serious contenders.
Feel free to send me any reports you get, especially if you find any unusual behavior or insane results. The autograntor isn't currently set up to evaluate any grants beyond standard AI-safety-related grants, and I don't expect it to perform well in any other field.
Why didn't I assign saving the world a value of one million points, implicitly pricing everything else in microdooms? I felt less certain that I could have an intuitive sense of how many microdooms the average alignment paper averts than that I could intuitively rank good grant outcomes relative to each other. Since on my model Plan A would decrease p(doom) by 20%pp, conversion to a microdoom based framework is trivial (within the model - if someone else has a different starting p(doom), they have to convert differently)
Subjectively - I'm not basing this on objective measures like money moved, because that depends on how much each grantee asked for.
If I'd been smart, I would have filled them in personally first, without AI help. But if I did this, I've since lost the file, and now I'm contaminated by knowing the AIs' values.
I also checked this by having an AI design itself a tournament where it showed itself two milestones at a time, rated how much better one was than another, and used an Elo-like algorithm to turn all of this into a set of consistent weights. This didn't change things much.
I think the rating of one point is indefensible on community-building grounds, but the whole point of this exercise is to quantify worldviews that differ from mine.
Again, something must be wrong with these numbers - a really great alignment paper is worth 5,000, so this suggests that the great alignment paper = convincing five million people, whereas common-sensically, convincing five million people is clearly better. But convincing every non-elite person in the US (let's say 10% of people are elites, so this is about 300 million), would be worth 300,000 points, or about a third of Plan A, which actually does seem order-of-magnitude right (if everyone were convinced, this would be a huge, but not completely sufficient, step to getting Plan A). My guess is either that this has something to do with double-counting, or that I'm failing to model elites-vs-normies correctly - it could be that on an individual level, elites are worth vastly more than normies, but convincing all normies at once means elites are likely to go along (or is simply nonsensical, since it's hard to imagine all normies but no elites being convinced).
Many of these grants are on Lightcone Commons. Thank you to Habryka and the Lightcone team for creating an AI connector that allows the autograntor to access the Lightcone Commons site directly. When a grant is on LC, the autograntor is able to read the application itself, the other evaluators' notes, and the transcripts of interviews with the grantees, which it takes seriously and adds to its estimates. I don't include any Lightcone-based analyses in the sample analysis section below, because I don't know what the privacy norms are and don't want to leak information from the applications. There's some risk that this access "contaminates" the autograntor, since it can see what humans (including me) thought. In my brief tests, this hasn't shifted its judgment noticeably.