mattmacdermott's Shortform

mattmacdermott

Coefficient Giving (where I work as a grantmaker) recently launched Project Tailwind, a call for founders of AI safety orgs^[1]. Many of the best existing AI safety orgs were started by LessWrong readers, and I expect some of the best orgs we fund through Tailwind will be too.

The website includes descriptions of 36 potential orgs that various CG grantmakers would like to fund, which I’ve pasted in a list below. But these are just examples – you can apply for funding to start^[2] any org that has a good shot at reducing catastrophic risk from AI. The strategic situation has been changing pretty fast, so there’s probably plenty of great org ideas that aren’t on this list.

CG now moves pretty fast and goes pretty big compared to historical baselines. For example, in June my team got approval for up to $160m to found Resolution in about 6 weeks. And in August we got a $10m grant to Jacob Tsimerman approved 12 days after he first reached out to us. The range of AI safety work that CG is willing to fund is pretty broad these days, too.

So funding isn’t really a bottleneck. But talent is! So if you think one of these orgs is a promising way to reduce x-risk, maybe you should be the person to make it happen. And if you think they’re all rubbish ideas, maybe you should apply with a better one. And if you don’t want to start an org, maybe you should post your better idea in the comments for someone else to use. It's also valuable to post critiques of these ideas.

We’re using a loose framework of pre-seed grants of $200k–$2m, seed grants of $2m–$20m, and bigger bets of $20m–$200m or even more. I expect that we’ll often give out pre-seed grants to promising people with good ideas, even when their plans are not particularly fleshed out and they expect to pivot around a lot. We also fund generous compute budgets and competitive salaries.

The easiest way to apply for funding is to fill out this quick expression-of-interest form. If your EOI sounds promising we’ll invite you to submit a full proposal.

Full list of example orgs:

Independent assessment

- Frontier lab accountability: Independent tracking, comparison, and review of frontier labs' safety-relevant practices.
- Misalignment red team: Discover misalignment properties in frontier models before the world finds out the hard way.
- Control red team: Stress-test AI companies' security and control protocols before a misaligned AI does.
- Bio safeguards red team: Systematic red teaming of gene synthesis and AIxBio safeguards.
- Independent AI auditors: Build the independent audit capacity necessary for AI regulation to succeed.
- Investigations for secret loyalties: Determine whether companies can prevent a small group of people from bending AI to their own interests.
- Misleadingness evaluations: Build evaluations that incentivize labs to make models less misleading on fuzzy tasks.

Research institutions

- Oversight of CoT monitorability: Measure how monitorable frontier model chains of thought are, track if that property degrades, and identify ways to preserve it.
- AI character evaluator: Improve our understanding of AIs' behavioral tendencies and overall values.
- Unclaimed research labs: Build a research lab around one of the many promising safety agendas still waiting for a team.
- Training run fault injections: Construct misaligned models in vitro to map which training procedures contribute to misalignment.
- Threat modeling institute: Develop and curate the best models of how catastrophic risks from AI could occur.
- Alignment moonshots: Pioneer alignment techniques robust enough to trust with superintelligent AI.

Evidence generation

- Rogue incident detection: Independent detection of real-world AI-driven attacks.
- AI incident investigations: Investigate reports of misalignment or misuse and determine their implications.
- Frontier capabilities observatory: Provide decision-makers with clear demonstrations of what frontier AI models can do.
- State of AI risk: Publish canonical living literature reviews on what we know about different risks from advanced AI.
- Biological uplift demos: Create and use demos to communicate frontier AI-bio uplift to policymakers.

AI security & verification

- Agreement verification technology: Build the technology and protocols to enable and verify agreements on safe AI development.
- Security capacity buildout: Build the talent pipelines and infrastructure to bring AI projects in the U.S. to SL4 and SL5.
- AI integrity standards: Build security protocols that provide assurance an AI's integrity has not been compromised.

Public goods

- Think tank for AI labs: On-demand independent expertise for frontier AI safety and governance teams.
- AI for decision-makers: Build AI tools that improve the quality of what could be the world's most important decisions.
- AI uplift for AI safety: Help the independent ecosystem make good use of frontier capabilities.
- Compute cluster for AI safety: Build and run a GPU cluster that serves the AI safety nonprofit ecosystem.
- Datasets for AI safety: Build datasets and environments that differentially improve AI safety capabilities.
- Grand awards for AI safety: Run competitions for important work in AI safety.

Talent acceleration

- Senior talent headhunting: Recruit senior or specialized talent to work on important areas in AI safety.
- Talent matchmaking: Build infrastructure to fill important roles in the AI safety ecosystem.
- Mid-career entry points: Expand opportunities for experienced professionals to join AI safety orgs.
- AI context accelerator: Help experienced professionals build the context and networks for AI safety leadership roles.
- Philanthropic startup incubator: Build the infrastructure to launch and scale cohorts of new AI safety projects.
- Fiscal sponsors: Build fiscal sponsorship capacity to absorb new organizations over the next 1-2 years.

Convenings

- Cross-sector convenings: Facilitate engagement among key groups working on frontier AI.
- Subfield forums: Run events aimed at focusing attention on specific areas within AI safety.
- Events as a service: Make sure every organization that wants to run an event can do so easily.

^{^}
Some of these initiatives might not require founding an org, and might be better described as projects, but I’ve used ‘org’ everywhere for simplicity.
^{^}
You can also apply for funding for an existing org.

mattmacdermott's Shortform

4

mattmacdermott's Shortform

4

Full list of example orgs: