Scott Alexander posted a survey of utility weights for a bunch of different AI safety accomplishments, discussed more in Towards the Autograntor.
The survey asked people to assign point values to AI safety milestones. The values were based on the anchor of 1 million points = a pause with China à la AI Futures Project’s Plan A, so a milestone assigned 10,000 points is 1% as good as that. Here’s a link to our survey responses.
To be clear, we (especially Austin) didn’t spend that much time on this and have low confidence in our answers! But we think being transparent about even half-baked opinions is good.
Headline numbers
As examples, here are a few milestones where the two of us (and the autograntor reflecting Scott’s worldview) gave relatively similar values:
Milestone
Austin
Caroline
Scott
Founding of effective altruism
100,000
75,000
100,000
Small-grant ecosystem (Long-Term Future Fund, SFF and similar)
20,000
20,000
30,000
Lighthaven
10,000
5,000
9,000
OpenAI or Anthropic becomes 1 point more alignment-pilled (on a scale where 0 = NVIDIA, 100 = Anthropic)
2,000
500
3,000
Austin’s and Caroline’s average ratings for alignment research and community-building milestones were similarish to each other’s, with Caroline rating policy outcomes more highly and Austin rating social shifts more highly.
This table shows the sum of our point values within categories, for rows we both scored:
Category
Austin
Caroline
Politics (rows scored by both)
62k
96k
Alignment (rows scored by both)
68k
58k
Community building (all rows)
354k
282k
Social shifts (all rows)
80k
32k
Within community building, Caroline gave higher ratings to earlier community-building efforts and Austin gave higher ratings to later ones. To some extent this seems like a credit allocation problem—how much credit should, say, the original LessWrong get for LW 2.0? On individual milestones, some rated very differently by the two of us were:
Milestone
Austin
Caroline
Comments
Problem framing: orthogonality, instrumental convergence, corrigibility, the sharp left turn
-10,000
60,000
Austin disagrees with this problem framing.
RLHF: reinforcement learning from human preferences, scaled up in InstructGPT
50,000
-10,000
Caroline rated it negatively because of its effects on speeding up AI progress.
Mandatory high security (SL4+) for frontier model weights
-10,000
6,000
Austin thinks weights should be open.
Restore presumption-of-denial export controls on frontier AI chips to China
-2,000
10,000
Austin thinks export controls are bad.
LessWrong and the Sequences
10,000
50,000
Example of allocating credit differently among early vs. late milestones.
LessWrong 2.0
40,000
5,000
see above
CAIS Statement on AI Risk
25,000
4,000
Pope Leo XIV makes AI a signature issue
3,000
300
Interesting disagreements
Why and how much do public shifts matter?
On the one hand, it seems like “public discourse on AI risk” has gone very well over the past few years. It’s become very much within the Overton window to talk about AI risk, and public opinion seems concerned about AI risk and pro-AI safety measures in a not-especially-polarized way.
On the other hand, what has this bought us?
It may have brought in a bit of funding and some new researchers, but most of the funding and the best research is still coming from people who have been alignment-pilled for a long time.
There have been some minor legislative victories, but mostly the public opinion hasn’t translated to major policy change. Labs are still racing toward RSI, and even if people are against it nothing is stopping them.
Austin pointed out the analogy of COVID. It took the world time to wake up to COVID, but once it did the reaction was very large-scale, with mass lockdowns across the world. He thinks we’ll see something like that for AI, and public opinion will create capacity that gets used for major countermeasures in a crisis.
How good (or bad) are secure weights and export controls?
Austin thinks frontier models should have open weights. He thinks bio and cyber risks are used as a specter when the crux of the AI companies is simply that they want to be in control of the most advanced models.
Caroline thinks they’re a real issue, not a specter, and lots of bad stuff would happen if frontier model weights were public by default, even at today’s capabilities and especially in the future.
On export controls, Austin is against—he prefers a world where access to advanced AI is widespread. Caroline thinks even regardless of the direct consequences of Chinese AI development, the farther ahead of China the US is, the more willing US labs will be to slow down for safety.
In general, there’s a question of the downsides of unipolarity vs multipolarity. Limiting access to powerful AI is undemocratic and risks concentration of power. In many domains, competition and a system of checks-and-balances is good. But also, the more people have AI, the more actors will be motivated to race ahead of others instead of slowing down for safety. Austin comes down on being more concerned about unipolarity, while Caroline ends up more concerned about multipolarity.
Funding
Scott wrote that he found the best projects to cost around $1-2k per point, decent projects $5-10k, and at greater than $20k per point a project wasn’t worth funding. We asked Claude to estimate costs for the milestones that have been achieved, and calculated their cost per point—you can see these on the “Cost per point” tab of our spreadsheet.
Again the numbers are quite rough and shouldn’t be taken too literally, but they vaguely line up with Scott’s guidelines. (Both Austin and Caroline valued milestones lower than Scott on average, making our cost-effectiveness somewhat lower.) Unsurprisingly, the best projects on this metric are past things that started small and turned out to be very impactful: milestones that came in at sub-$100 a point include the founding of EA, HPMOR, Less Wrong and the Sequences, and the existence of a Berkeley rationalist hub.
There were also more recent milestones we both rated as very cost-effective: the CAIS statement on AI risk ($4-25 per point), AI impacts surveys ($70-700), AI 2027 ($150-500), and the FLI pause letter ($200-400).
Our numbers suggest giving an extra $1b from Coefficient Giving or SFF according to their usual procedures is quite cost-ineffective ($60-600k per point). These values could well be an overestimate, but we do think that giving away billions more dollars effectively requires more diversity of funders and funding structures than just trying to expand what already exists.
What next?
We’re excited for more people to take the survey, and share their values or comment on why ours are wrong—we think this exercise is a cool way to quantify disagreements in AI safety.
In general we’re excited to see more AI-powered grantmaking of the kind Scott proposes—see also Using AI to find good Manifund projects. As the money in the ecosystem grows, it seems like grantmaker capacity will be a big bottleneck in giving it away unless we can find ways to automate parts of the process.
Scott Alexander posted a survey of utility weights for a bunch of different AI safety accomplishments, discussed more in Towards the Autograntor.
The survey asked people to assign point values to AI safety milestones. The values were based on the anchor of 1 million points = a pause with China à la AI Futures Project’s Plan A, so a milestone assigned 10,000 points is 1% as good as that. Here’s a link to our survey responses.
To be clear, we (especially Austin) didn’t spend that much time on this and have low confidence in our answers! But we think being transparent about even half-baked opinions is good.
Headline numbers
As examples, here are a few milestones where the two of us (and the autograntor reflecting Scott’s worldview) gave relatively similar values:
Milestone
Austin
Caroline
Scott
Founding of effective altruism
100,000
75,000
100,000
Small-grant ecosystem (Long-Term Future Fund, SFF and similar)
20,000
20,000
30,000
Lighthaven
10,000
5,000
9,000
OpenAI or Anthropic becomes 1 point more alignment-pilled (on a scale where 0 = NVIDIA, 100 = Anthropic)
2,000
500
3,000
Austin’s and Caroline’s average ratings for alignment research and community-building milestones were similarish to each other’s, with Caroline rating policy outcomes more highly and Austin rating social shifts more highly.
This table shows the sum of our point values within categories, for rows we both scored:
Category
Austin
Caroline
Politics (rows scored by both)
62k
96k
Alignment (rows scored by both)
68k
58k
Community building (all rows)
354k
282k
Social shifts (all rows)
80k
32k
Within community building, Caroline gave higher ratings to earlier community-building efforts and Austin gave higher ratings to later ones. To some extent this seems like a credit allocation problem—how much credit should, say, the original LessWrong get for LW 2.0? On individual milestones, some rated very differently by the two of us were:
Milestone
Austin
Caroline
Comments
Problem framing: orthogonality, instrumental convergence, corrigibility, the sharp left turn
-10,000
60,000
Austin disagrees with this problem framing.
RLHF: reinforcement learning from human preferences, scaled up in InstructGPT
50,000
-10,000
Caroline rated it negatively because of its effects on speeding up AI progress.
Mandatory high security (SL4+) for frontier model weights
-10,000
6,000
Austin thinks weights should be open.
Restore presumption-of-denial export controls on frontier AI chips to China
-2,000
10,000
Austin thinks export controls are bad.
LessWrong and the Sequences
10,000
50,000
Example of allocating credit differently among early vs. late milestones.
LessWrong 2.0
40,000
5,000
see above
CAIS Statement on AI Risk
25,000
4,000
Pope Leo XIV makes AI a signature issue
3,000
300
Interesting disagreements
Why and how much do public shifts matter?
On the one hand, it seems like “public discourse on AI risk” has gone very well over the past few years. It’s become very much within the Overton window to talk about AI risk, and public opinion seems concerned about AI risk and pro-AI safety measures in a not-especially-polarized way.
On the other hand, what has this bought us?
It may have brought in a bit of funding and some new researchers, but most of the funding and the best research is still coming from people who have been alignment-pilled for a long time.
There have been some minor legislative victories, but mostly the public opinion hasn’t translated to major policy change. Labs are still racing toward RSI, and even if people are against it nothing is stopping them.
Austin pointed out the analogy of COVID. It took the world time to wake up to COVID, but once it did the reaction was very large-scale, with mass lockdowns across the world. He thinks we’ll see something like that for AI, and public opinion will create capacity that gets used for major countermeasures in a crisis.
How good (or bad) are secure weights and export controls?
Austin thinks frontier models should have open weights. He thinks bio and cyber risks are used as a specter when the crux of the AI companies is simply that they want to be in control of the most advanced models.
Caroline thinks they’re a real issue, not a specter, and lots of bad stuff would happen if frontier model weights were public by default, even at today’s capabilities and especially in the future.
On export controls, Austin is against—he prefers a world where access to advanced AI is widespread. Caroline thinks even regardless of the direct consequences of Chinese AI development, the farther ahead of China the US is, the more willing US labs will be to slow down for safety.
In general, there’s a question of the downsides of unipolarity vs multipolarity. Limiting access to powerful AI is undemocratic and risks concentration of power. In many domains, competition and a system of checks-and-balances is good. But also, the more people have AI, the more actors will be motivated to race ahead of others instead of slowing down for safety. Austin comes down on being more concerned about unipolarity, while Caroline ends up more concerned about multipolarity.
Funding
Scott wrote that he found the best projects to cost around $1-2k per point, decent projects $5-10k, and at greater than $20k per point a project wasn’t worth funding. We asked Claude to estimate costs for the milestones that have been achieved, and calculated their cost per point—you can see these on the “Cost per point” tab of our spreadsheet.
Again the numbers are quite rough and shouldn’t be taken too literally, but they vaguely line up with Scott’s guidelines. (Both Austin and Caroline valued milestones lower than Scott on average, making our cost-effectiveness somewhat lower.) Unsurprisingly, the best projects on this metric are past things that started small and turned out to be very impactful: milestones that came in at sub-$100 a point include the founding of EA, HPMOR, Less Wrong and the Sequences, and the existence of a Berkeley rationalist hub.
There were also more recent milestones we both rated as very cost-effective: the CAIS statement on AI risk ($4-25 per point), AI impacts surveys ($70-700), AI 2027 ($150-500), and the FLI pause letter ($200-400).
Our numbers suggest giving an extra $1b from Coefficient Giving or SFF according to their usual procedures is quite cost-ineffective ($60-600k per point). These values could well be an overestimate, but we do think that giving away billions more dollars effectively requires more diversity of funders and funding structures than just trying to expand what already exists.
What next?
We’re excited for more people to take the survey, and share their values or comment on why ours are wrong—we think this exercise is a cool way to quantify disagreements in AI safety.
In general we’re excited to see more AI-powered grantmaking of the kind Scott proposes—see also Using AI to find good Manifund projects. As the money in the ecosystem grows, it seems like grantmaker capacity will be a big bottleneck in giving it away unless we can find ways to automate parts of the process.