Jesse Li's Shortform

Jesse Li

This is a special post for quick takes by Jesse Li. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Would an "Unsupervised Program for Alignment Research" be effective?

I think there may be a substantial pool of people trying to contribute to AI safety who are motivated, talented, and capable of doing good independent research, who nonetheless haven't been accepted to any existing mentored fellowship.

Synthesizing a few perspectives I've heard, which I don't think are necessarily contradictory:

There are not enough people working full time in AI safety compared to the scale of the problem: a few thousand FTEs; contrast with e.g. the Federal Aviation Administration (FAA) which employs >40k people.
There's a lot of low-hanging fruit with respect to AI research and the field could use more diversity in ideas.
There are too many people trying to transition into AI safety work. Fellowships are extremely competitive.
A substantial fraction of the applicants/applications to a given program don't meet the bar (AI generated answers, insufficient technical skill, poor evidence of ability to do independent research).
However, even after you filter for high-quality applicants, there's still too many compared to the available mentors to manage them (maybe 2~3x).
Funding is bottlenecked by the number of high quality projects, not money.

Note: I'm a SPAR mentee and have only been on the applying side of these programs, so I don't know how accurate the above claims are. But they do seem to line up with what I've seen mentors say.

So, what if we just found these people & gave them funding anyway, to do independent research?

What I'm imagining is a part-time program like SPAR, but with a lower demand on mentors: Only that they submit a research question they'd like answered.
Applicants would be selected for high demonstrated ability to work without supervision.
Program attendees could work in teams so they could get feedback from each other. To compensate for the limited senior supervision, there could be regular structured and unstructured methods for different teams to interact with each other to provide additional feedback.

2Brendan Long3d

I think finding ways to scale up mentorship better is a good idea, but I worry that your proposal is mostly just SPAR. The mentors already aren't expected to do research and technically only have to have a research question and meet with mentees every week or so. Even with that, my SPAR mentor spends a significant amount of time explaining which aspects of the research question are interesting, where other people will push back, and what's happening in the field that affects what we should be doing. It does feel like there should be ways to scale this further to reduce the demands on mentors though. Part of the complexity seems to be from the size of projects. A set of micro-experiments that mentees should run and then write up would be interesting (although it might be hard to find something small enough that Claude can't do on its own).

Valid concern; my proposal is literally just putting "un-" in front of "Supervised Program for Alignment Research". In order to do something SPAR isn't already, I think you'd have to reduce mentor involvement requirements by an order of magnitude:

At the application review phase, don't require mentors to review applications. You'd have to instead primarily evaluate candidates centrally on general technical/research skill. Perhaps there's a way to have candidates evaluate each other, for scalable application review? For example, each project could require ... (read more)

Jesse Li's Shortform

1

Jesse Li's Shortform

1