Ryan Kidd's Shortform

Ryan Kidd

This is a special post for quick takes by Ryan Kidd. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Why does the AI safety community need help founding projects?

AI safety should scale
1. Labs need external auditors for the AI control plan to work
2. We should pursue many research bets in case superalignment/control fails
3. Talent leaves MATS/ARENA and sometimes struggles to find meaningful work for mundane reasons, not for lack of talent or ideas
4. Some emerging research agendas don’t have a home
5. There are diminishing returns at scale for current AI safety teams; sometimes founding new projects is better than joining an existing team
6. Scaling lab alignment teams are bottlenecked by management capacity, so their talent cut-off is above the level required to do “useful AIS work”
Research organizations (inc. nonprofits) are often more effective than independent researchers
1. “Block funding model” is more efficient, as researchers can spend more time researching, rather than seeking grants, managing, or other traditional PI duties that can be outsourced
2. Open source/collective projects often need a central rallying point (e.g., EleutherAI, dev interp at Timaeus, selection theorems and cyborgism agendas seem too delocalized, etc.)
There is (imminently) a market for for-profit AI safety companies and value-aligned people should capture this free energy or let worse alternatives flourish
1. If labs or API users are made legally liable for their products, they will seek out external red-teaming/auditing consultants to prove they “made a reasonable attempt” to mitigate harms
2. If government regulations require labs to seek external auditing, there will be a market for many types of companies
3. “Ethical AI” companies might seek out interpretability or bias/fairness consultants
New AI safety organizations struggle to get funding and co-founders despite having good ideas
1. AIS researchers are usually not experienced entrepeneurs (e.g., don’t know how to write grant proposals for EA funders, pitch decks for VCs, manage/hire new team members, etc.)
2. There are not many competent start-up founders in the EA/AIS community and when they join, they don’t know what is most impactful to help
3. Creating a centralized resource for entrepeneurial education/consulting and co-founder pairing would solve these problems

4Ryan Kidd2y

I interpret your comment as assuming that new researchers with good ideas produce more impact on their own than in teams working towards a shared goal; this seems false to me. I think that independent research is usually a bad bet in general and that most new AI safety researchers should be working on relatively few impactful research directions, most of which are best pursued within a team due to the nature of the research (though some investment in other directions seems good for the portfolio). I've addressed this a bit in thread, but here are some more thoughts: * New AI safety researchers seem to face mundane barriers to reducing AI catastrophic risk, including funding, infrastructure, and general executive function. * MATS alumni are generally doing great stuff (~78% currently work in AI safety/control, ~1.4% work on AI capabilities), but we can do even better. * Like any other nascent scientific/engineering discipline, AI safety will produce more impactful research with scale, albeit with some diminishing returns on impact eventually (I think we are far from the inflection point, however). * MATS alumni, as a large swathe of the most talented new AI safety researchers in my (possibly biased) opinion, should ideally not experience mundane barriers to reducing AI catastrophic risk. * Independent research seems worse than team-based research for most research that aims to reduce AI catastrophic risk: * "Pair-programming", builder-breaker, rubber-ducking, etc. are valuable parts of the research process and are benefited by working in a team. * Funding insecurity and grantwriting responsibilities are larger for independent researchers and obstruct research. * Orgs with larger teams and discretionary funding can take on interns to help scale projects and provide mentorship. * Good prosaic AI safety research largely looks more like large teams doing engineering and less like lone geniuses doing maths. Obviously, some lone genius researchers (espec

I interpret your comment as assuming that new researchers with good ideas produce more impact on their own than in teams working towards a shared goal

I don't believe that, although I see how my summary could be interpreted that way. I agree with basically all the reasons in your recent comment and most in the original comment. I could add a few reasons of my own doing independent grant-funded work sucks. But I think it's really important to track how founding projects tracks to increased potential safety instead of intermediates, and push hard against pote... (read more)

4Ryan Kidd2y

Also note that historically many individuals entering AI safety seem to have been pursuing the "Connector" path, when most jobs now (and probably in the future) are "Iterator"-shaped, and larger AI safety projects are also principally bottlenecked by "Amplifiers". The historical focus on recruiting and training Connectors to the detriment of Iterators and Amplifiers has likely contributed to this relative talent shortage. A caveat: Connectors are also critical for founding new research agendas and organizations, though many self-styled Connectors would likely substantially benefit as founders by improving some Amplifier-shaped soft skills, including leadership, collaboration, networking, and fundraising.