This is an executive summary of a post from my personal blog, also cross-posted from the EA Forum. Read the full texts here.
Summary
Benchmarks support the empirical, quantitative evaluation of progress in AI research. Although benchmarks are ubiquitous in most subfields of machine learning, they are still rare in the subfield of AI safety.
I argue that creating benchmarks should be a high priority for AI safety. While this idea is not new, I think it may still be underrated. Among other benefits, benchmarks would make it much easier to:
track the field’s progress and focus resources on the most productive lines of work;
create professional incentives for researchers - especially Chinese researchers - to work on problems that are relevant to AGI safety;
develop auditing regimes and regulations for advanced AI systems.
Unfortunately, we cannot assume that good benchmarks will be developed quickly enough “by default." I discuss several reasons to expect them to be undersupplied. I also outline actions that different groups can take today to accelerate their development.
For example, AI safety researchers can help by:
directly trying their hand at creating safety-relevant benchmarks;
clarifying certain safety-relevant traits (such as “honesty” and “power-seekingness”) that it could be important to measure in the future;
building up relevant expertise and skills, for instance by working on other benchmarking projects;
drafting “benchmark roadmaps,” which identify categories of benchmarks that could be valuable in the future and outline prerequisites for developing them.
And AI governance professionals can help by:
co-organizing workshops, competitions, and prizes focused on benchmarking;
creating third-party institutional homes for benchmarking work;
clarifying, ahead of time, how auditing and regulatory frameworks can put benchmarks to use;
advising safety researchers on political, institutional, and strategic considerations that matter for benchmark design;
popularizing the narrative of a “race to the top” on AI safety.
Ultimately, we can and should begin to build benchmark-making capability now.
Acknowledgment
I would like to thank Ben Garfinkel and Owen Cotton-Barratt for their mentorship, Emma Bluemke and many others at the Centre for the Governance of AI for their warmhearted support. All views and errors are my own.
Future research
I am working on a paper on the topic, and if you are interested in benchmarks and model evaluation, especially if you are a technical AI safety researcher, I would love to hear from you!
I didn't. I'm sure words towards articulating this have been spoken many times, but the trick is in what forum / form does it need to exist more specifically in order for it to be comprehensible and lasting. Maybe I'm wrong that it needs to be highly public; as with nukes not many people are actually familiar with what is considered sufficient fissile material - governments (try to) maintain this barrier by themselves. But at this stage as it still seems a fuzzy concept, any input seems valid.
Consider the following combination of properties:
(software - if that's the right word?) capable of self replication / sustainability / improvement
capable of eluding human control
capable of doing harm
In isolation none of these is sufficient, but taken together I think we could all agree we have a problem. So we could begin to categorize and rank various assemblages of AI by these criteria, and not by how "smart" they are.
This is an executive summary of a post from my personal blog, also cross-posted from the EA Forum. Read the full texts here.
Summary
Benchmarks support the empirical, quantitative evaluation of progress in AI research. Although benchmarks are ubiquitous in most subfields of machine learning, they are still rare in the subfield of AI safety.
I argue that creating benchmarks should be a high priority for AI safety. While this idea is not new, I think it may still be underrated. Among other benefits, benchmarks would make it much easier to:
Unfortunately, we cannot assume that good benchmarks will be developed quickly enough “by default." I discuss several reasons to expect them to be undersupplied. I also outline actions that different groups can take today to accelerate their development.
For example, AI safety researchers can help by:
And AI governance professionals can help by:
Ultimately, we can and should begin to build benchmark-making capability now.
Acknowledgment
I would like to thank Ben Garfinkel and Owen Cotton-Barratt for their mentorship, Emma Bluemke and many others at the Centre for the Governance of AI for their warmhearted support. All views and errors are my own.
Future research
I am working on a paper on the topic, and if you are interested in benchmarks and model evaluation, especially if you are a technical AI safety researcher, I would love to hear from you!