I recently served as a recommender in SFF's annual funding round (grants will be decided and announced in September). I'm deeply grateful for SFF's funders, and I hope more AI safety donors appear in the future.

Unfortunately, the SFF experience is bad for both applicants and recommenders, it's slow, and it lacks some other desiderata. (The actual funding decisions are moderately good by my lights — probably over half of the value of grants that a better system with the same budget would produce, but substantial room for improvement.) The basic idea of the s-process—several recommenders compete to convince the funders to fund their recommendations each round, and so several different perspectives each get the best things by their lights funded and there's little delegation of power—is attractive. But SFF's implementation is bad.

My experience as a recommender

• Recommenders were told we would only have to spend 30 hours, but the stuff we were required to do took much more than 30 hours, and doing a good job took even longer.^[1]
  • [Cherry-picked / unusually legible] example suggesting that the time commitment was more than SFF said: we were asked to spend one hour glancing at the top of some 150-200 applications and, for each application, score our interest in investigating from 0 to 4. This was not feasible (and it was initially impossible to complete due to software bugs).
• The custom software (to let recommenders process applications and make recommendations) was bad — it had lots of bugs and it just failed to make evaluating applications easy.
  • The information that the software thinks I want isn't the information that I actually want.
  • Navigating within and (especially) between applications was hard.
  • The application-review workflow was just annoying/hard/clunky in many ways.
  • Each application was given an AI-generated name; many were convoluted, confusing, ambiguous, and/or incorrect. (Overall substantially worse than the names provided by the applicants; often the applicant-provided names were hidden so we could only see the bad AI names.)
  • Because applications are so long, I think I read zero applications in full. Recommenders often engaged with AI summaries of applications. This is inefficient; it would be better in many ways if applications were shorter and recommenders read the applications.
  • Usually my changes were autosaved but regularly they were just deleted.
  • Several more I'm forgetting.^[2]
• Many recommenders were ill-equipped for the task in various ways — lacking experience, lacking info on grantmaking, misunderstanding the utility functions they create.
  • Rather than recommending a certain grant size to each applicant, recommenders create a utility function for each applicant representing the value of giving them various amounts of money, and the utility functions automatically determine grant sizes. I believe there's multiple cases where, at least before I intervened, a recommender's utility functions clearly didn't represent their views. For example, one (smart, experienced) recommender's utility functions initially said that for each applicant, marginal funding hits zero value by $1.7M to the applicant; after I expressed disagreement, they changed their curves to show that some applicants would produce positive value with marginal funding beyond $1.7M. Unfortunately most issues are more subtle.
  • Some issues could be improved by choosing different recommenders. Some issues could be improved by offering better resources to recommenders.
  • Several recommenders were predictably bad choices by my lights. (In theory it could fine if an s-process has some bad recommenders—in theory the funders can just disregard them—but I expect recommender selection is important for SFF.)
• Lack of division of labor. Every recommender evaluates every application (skipping ones they know they're not interested in), and you can see other recommenders' notes and ask them questions but overall division-of-labor mechanisms were poor.
• Lack of institutional knowledge, lack of info on particular orgs, and other structural-ish problems.
  • For structural reasons there's nobody responsible for paying attention to an applicant year-to-year. A more solvable problem is that by default recommenders don't have access to info that professional grantmakers have about various orgs. Also most recommenders have never done this before (like me!) and there's little training/resources.
• Applicants can ask for a matching pledge in addition to a grant. This is a fine idea but the implementation is a mess.^[3] And I'd guess most of the recommenders don't really understand how it works (and details were only explained to us at the very end of the process) and this will lead to funding recommendations that don't match the recommenders' views.

My impression is that other recommenders also had a bad experience. If SFF surveys the six main track recommenders in the next few weeks, for the five besides me, I predict at least three will say they had an overall negative experience (but most others are less pessimistic than me).

Applicant experience

The application is exceptionally long and annoying; I think SFF is an outlier in this dimension; it causes those who do apply to waste time and other promising projects to not apply at all. The process is slow: it takes five months from application deadline to funding decisions (and it only runs once per year) (but some small projects can be funded quickly by "speculation granters"). And these days to be considered you have to submit another application explaining why you need money quickly (even if you don't) and receive a "speculation grant" based on that application.

Some desiderata

• Making good grants: meh.^[4]
  • Part of this is real disagreements between me and other recommenders and/or funders, but a big part is structural stuff including "SFF recommenders are insufficiently sensitive to applicants' runways or funging with GV^[5] and inflexible funders" (since the software doesn't make this salient and recommenders aren't trained to care about it), issues with recommenders not knowing how to express their views in the software (especially for matching pledges), recommenders lacking some kinds of info, maybe "SFF isn't set up to deal with adverse selection," and many of the aforementioned issues with the process (which just make recommenders' recommendations less good).
• Speed: poor.
  • Five months from application deadline to funding decisions. Some orgs don't need funding quickly, but in many cases a project needs funding—e.g. runway for at least a year—in order to start or to scale up. Also facts and plans change in five months.
• Flexibility: poor.
  • You can't do private stuff, you can't really coordinate with other funders, you can't really steer grantees. See Considerations against s-process philanthropy for details on some desiderata — some of those issues are surmountable, but SFF doesn't currently surmount them.
• Predictability: meh to poor. (Less important, especially since SFF is a small fraction of total funding.)
  • Almost all applicants are pretty unable to predict how much funding they'll receive. For structural reasons (including that the total budget and distribution between recommenders is only determined at the very end of the process), even recommenders have little sense of how much funding each applicant will receive. To some extent this is inevitable in a system designed to avoid delegation of power from the funder to the recommenders.
  • SFF has recently tried implementing some confusing policies on applicant eligibility, including policies aimed at avoiding funding a large fraction of "advocacy" projects. This has costs including wasting evaluation time and causing uncertainty/chaos for various groups of people in addition to the prospect of missing good funding opportunities.
• (SFF, as an s-process, does seem to succeed at avoiding large-scale delegation of power (which results in various problems) and keeping the philanthropy responsive to the funders' views.)

I'm not asserting that SFF is the worst AI safety funding institution. And relative to a vacuum—that is, if not for the effect of crowding out potential similar efforts—I believe SFF is substantially net-positive. I just wrote this post because I recently developed conviction that SFF is leaving lots of cheap value on the table.

This post is partially written as feedback for SFF, but I have little faith in SFF to improve. The status quo seems to be that SFF isn't trying to be good (e.g. it had recommenders use software and processes that were clearly untested, and it apparently doesn't do user interviews). So I think specific suggestions won't go far without a general wakeup call.

habryka will soon launch a new s-process system. I am optimistic that it will be much better than SFF. If it is, I hope SFF's funders switch to it.

1. ^{^}

   I expected from the beginning that most recommenders would spend much more than 30 hours. I incorrectly thought I personally would be able to spend just 30 hours. I still think I could have made good recommendations in 30 hours, but (1) we were required to do specific things which I wouldn't have done if I was trying to do my best in 30 hours (e.g. enter an evaluation for at least 50 applications) and (2) your budget is determined at the end of the process and is indirectly correlated with how much time you spend so if you want to have more impact you have to spend more time.

2. ^{^}

   After writing this post I glanced at the relevant slack channel and yep, there's much more. And glancing at that channel reminded me of some bugs that weren't reported in that channel, including:

   • Text-editor tools (bullets, bold, italics) didn't work in my browser
   • Applications with special characters in the name initially couldn't be opened.
3. ^{^}

   Claim not justified here but my impression is almost everyone agrees. I think several—perhaps most/all—recommenders' views aren't captured by their utility functions for matching pledges. For one, I expect several recommenders left the parameters at their default values, which may be systematically wrong. And applicants can't really predict the implications of opting in.

4. ^{^}

   To be clear, the funders haven't yet decided what to fund this round. This is based on past rounds, my guesses about this round, and just a priori reason from how the process works.

5. ^{^}

   "Funge with" (or "funge against") means substitute with or displace or be displaced by. "GV" means Good Ventures. In many cases, GV is happy for flexible AI safety funders to not donate to the orgs it fully funds (and instead donate to the good projects that it doesn't fund for various reasons).