LESSWRONG
LW

AI EvaluationsAI

74

100+ concrete projects and open problems in evals

by Marius Hobbhahn

22nd Mar 2025

AI Alignment Forum

1 min read

74

Ω 39

AI EvaluationsAI

74

Ω 39

100+ concrete projects and open problems in evals

New Comment

1 comment, sorted by

Click to highlight new comments since: Today at 6:14 AM

[-]Alex Lawsen 4mo90

Thanks for putting this list together. It seems like a great resource for people interested in the field and looking for somewhere to start.

I think many of the projects described within it seem likely to be within scope for this request for proposals. If you're reading this and are excited about taking on one of these projects but need funding to do so, I encourage you to apply!

Curated and popular this week

We made a long list of concrete projects and open problems in evals with 100+ suggestions!

https://docs.google.com/document/d/1gi32-HZozxVimNg5Mhvk4CvW4zq8J12rGmK_j2zxNEg/edit?usp=sharing

We hope that makes it easier for people to get started in the field and to coordinate on projects.

Over the last 4 months, we collected contributions from 20+ experts in the field, including people who work at Apollo, METR, Redwood, RAND, AISIs, frontier labs, SecureBio, AI futures project, many academics, and independent researchers (suggestions have not necessarily been made in an official capacity). The doc has comment access, and further well-intentioned contributions are welcome!

Here is a screenshot of the table of contents: