Navigating AI safety fellowships and other programs as an early researcher is confusing because there are many of them, and it's not obvious which to prioritize or in what order. This post gives a rough progression path.

For this post, we assume the goal is to get a full-time job as a technical AI safety researcher. Fellowships and other programs help achieve this goal in three ways: 1) upskilling, 2) gaining career capital, and 3) networking in AI safety community.

Let me lay some background on skills, career capital, and networking, and after that, my thoughts on how to navigate AI safety programs.

Background

Skills

We can roughly divide technical AI safety research into two categories – empirical and theoretical. Empirical research is mostly about hands-on experiments with transformers, and theoretical research is essentially mathematical research. The majority (I would estimate >80%) of full-time AI safety researchers do empirical research, and the majority of teams on fellowships do empirical research, so I will concentrate on that.

Empirical research. A non-senior empirical researcher needs two main sets of skills:

General research – ability to understand research papers, generate interesting questions, design experiments to answer these questions, interpret the results, notice contradictions and missing parts.
Engineering – ability to write code that works, conduct experiments, set infrastructure for these experiments.

You can see fellowships through the lens of upskilling via these two axes. There are two corresponding archetypes of people who participate in fellowships:

Researchers from other science areas. They have several years of experience researching physics/math/cognitive science or other sciences, but have little coding experience. They focus on improving their engineering ability with transformers.
Engineers without research experience. They have several years of software development experience. They either already worked with transformers or quickly learn transformer-related engineering. They focus on improving research skills.

Note that if you already have both engineering skill and general research skill – for instance, you are a machine learning researcher with several publications – then you probably should apply only for the top fellowships (like MATS) or straight to the full-time researcher jobs.

Career capital

Fellowships and other programs g...

Kenneth Sun

Kenneth Sun