Aren't the central example of founders in AI Safety the people who founded Anthropic, OpenAI and arguably Deepmind? Right after that Mechanize comes to mind. 

I am not fully sure what you mean by founders, but it seems to me that the best organizations were founded by people who also wrote a lot, and generally developed a good model of the problems in parallel to running an organization. Even this isn't a great predictor. I don't really know what is. It seems like generally working in the space is just super high variance.

To be clear, overall I do think many more people should found organization, but the arguments in this post seem really quite weak. The issue is really not that otherwise we "can't scale the AI Safety field". If anything it goes the other way around! If you just want to scale the AI safety field, go work at one of the existing big organizations like Anthropic, or Deepmind, or Far Labs or whatever. They can consume tons of talent, and you can probably work with them on capturing more talent (of course, I think the consequences of doing so for many of those orgs would be quite bad, but you don't seem to think so).

Also, to expand some more on your coverage of coun... (read more)

I want to register disagreement. Multiplier effects are difficult to get and easy to overestimate. It's very difficult to get other people working on the right problem, rather than slipping off and working on an easier but ultimately useless problem. From my perspective, it looks like MATS fell into this exact trap. MATS has kicked out ~all the mentors who were focused on real problems (in technical alignment) and has a large stack of new mentors working on useless but easy problems.

[Edit 5hrs later: I think this has too much karma because it's political and aggressive. It's a very low effort criticism without argument.]

I think that being a good founder in AI safety is very hard, and generally only recommend doing it after having some experience in the field - this strongly applies to research orgs, but also to eg field building. If you're founding something, you need to constantly make judgements about what is best, and don't really have mentors to defer to, unlike many entry level safety roles, and often won't get clear feedback from reality if you get them wrong. And these are very hard questions, and if you don't get them right, there's a good chance your org is mediocre. I think this applies even to orgs within an existing research agenda (most attempts to found mech interp orgs seem doomed to me). Field building is a bit less dicey, but even then, you want strong community connections and a sense for what will and will not work.

I'm very excited for there to be more good founders in AI Safety, but don't think loudly signal boosting this to junior people is a good way to achieve this. And imo "founding an org" is already pretty high status, at least if you're perceived to have some momentum behind you?

I'm also fine with people without a lot of AI safety expertise partnering with those who do have it as co founders, but I struggle to think of orgs that I think have gone well who didn't have at least one highly experienced and competent co-founder

Great founders and field-builders have multiplier effects on recruiting, training, and deploying talent to work on AI safety [...] If we want to 10-100x the AI safety field in the next 8 years, we need multiplicative capacity, not just marginal hires

I spent much of 2018-2020 trying to help MIRI with recruiting at AIRCS workshops. At the time, I think AIRCS workshops and 80k were probably the most similar things the field had to MATS, and I decided to help with them largely because I was excited about the possibility of multiplier effects like these.

The single most obvious effect I had on a participant—i.e., where at the beginning of our conversations they seemed quite uninterested in working on AI safety, but by the end reported deciding to—was that a few months later they quit their (non-ML) job to work on capabilities at OpenAI, which they have been doing ever since.

Multiplier effects are real, and can be great; I think AIRCS probably had helpful multiplier effects too, and I'd guess the workshops were net positive overall. But much as pharmaceuticals often have paradoxical effect—i.e., to impact the intended system in roughly the intended way, except with the sign of the key eff... (read more)

I don't think this captures the counterarguments well. So here is one

You can imagine a spectrum of funders where on one hand, you have people who understand themselves as funders and want to be marshaling an army to solve AI alignment. On the other side, you have basically researchers who see work that should be done, don't have capacity to do the work themselves, and this leads them to create teams and orgs - "reluctant founders". 

It's reasonable to be skeptical about what the "funder type" end of the spectrum will do. 

In normal startups, the ultimate feedback loop is provided by the market. In AI safety nonprofits, the main feedback loops are provided by funders, AGI labs, and Bay Area prestige gradients. 
Bay Area prestige gradients are to a large extent captured by AGI labs - the majority of quality-weighted "AI safety" already works there, the work is "obviously impactful", you are close to the game, etc. also normal ML people also want to work there. 
If someone wants to scale a lot, "funders" means mostly OpenPhil - no other source would fund the army. The dominant OpenPhil worldview is closely related to Anthropic - for example, until recently you have hea... (read more)

I agree the AI safety field in general vastly undervalues building things, especially compared to winning intellectual status ladders (e.g. LessWrong posting, passing the Anthropic recruiting funnel, etc.).

However, as I've written before:

[...] the real value of doing things that are startup-like comes from [...] creating new things, rather than scaling existing things [...]

If you want to do interpretability research in the standard paradigm, Goodfire exists. If you want to do evals, METR exists. Now, new types of evals are valuable (e.g. Andon Labs & vending bench). And maybe there's some interp paradigm that offers a breakthrough.

But why found? Because there is a problem where everyone else is dropping the ball, so there is no existing machine where you can turn the crank and get results towards that problem.

Now of course I have my opinions on where exactly everyone else is dropping the ball. But no doubt there are other things as well.

To pick up the balls, you don't start the 5th evals company or the 4th interp lab. My worry is that that's what all the steps listed in "How to be a founder" point towards. Incubators, circulating pitches, asking for feedback on ideas, applying ... (read more)

This is my first Lesswrong comment - any feedback appreciated.

My quick takes (with a similar conflict: I'm doing AIS field-building).

1. I am inclined to agree with ~everything in this post.
2. I think the status dynamics are hard to overstate.
   1. I know quite a few very competent builders / 'doers' who have bounced off EA/AIS.
      1. And part of this is about the elevated status given to researchers, especially in contrast with the way 'operations' people (a catch-all used to encompass a large fraction of everything else) are treated.
   2. The response I often hear, explicitly or in the undertones, is: 'But if they were really committed, they'd just do the thing that needs to be done.' So they are expected to ignore the status gradients and e.g. build a nonprofit that is illegible to those outside EA/AIS or a for-profit with a harder path to profitability.
   3. Meanwhile, these communities are usually excited about people doing relevant-seeming AIS research, even when they might be doing so because it's interesting or high-status, rather than because it's the thing to do that has highest impact or is most important to AIS.
   4. I think (c) is usually good - pain is not the unit of effort and people are usually more pr

I don't think I quite understand the distinction you are trying to draw between "founders" and (not a literal quote) "people who do object-level work and make intellectual contributions by writing".

If you're the CEO of a company, it's your job to understand the space your company works in and develop extremely good takes about where the field is going and what your company should do, and use your expertise in leveraged ways to make the company go better.

In the context of AI safety, the key product that organizations are trying to produce is often itself re... (read more)

Hey Ryan, nice post. Here are some thoughts.

Anti-correlated attributes: “Founder‑mode” is somewhat anti‑natural to “AI concern.” The cognitive style most attuned to AI catastrophic risk (skeptical, risk‑averse, theory-focused) is not the same style that woos VCs, launches companies, and ships MVPs. If we want AI safety founders, we need to counterweight the selection against risk-tolerant cognitive styles to prevent talent drift and attract more founder-types to AI safety.

I think AI safety founders should be risk-averse.

For-profit investors like risk-seeki... (read more)