We agree, which is why we note, "We think that ~1 more median MATS scholar focused on AI safety is worth 5-10 more median capabilities researchers (because most do pointless stuff like image generation, and there is more low-hanging fruit in safety)."

MATS' goals:

• Find + accelerate high-impact research scholars:
  • Pair scholars with research mentors via specialized mentor-generated selection questions (visible on our website);
  • Provide a thriving academic community for research collaboration, peer feedback, and social networking;
  • Develop scholars according to the “T-model of research” (breadth/depth/epistemology);
  • Offer opt-in curriculum elements, including seminars, research strategy workshops, 1-1 researcher unblocking support, peer study groups, and networking events;
• Support high-impact research mentors:
  • Scholars are often good research assistants and future hires;
  • Scholars can offer substantive new critiques of alignment proposals;
  • Our community, research coaching, and operations free up valuable mentor time and increase scholar output;
• Help parallelize high-impact AI alignment research:
  • Find, develop, and refer scholars with strong research ability, value alignment, and epistemics;
  • Use alumni for peer-mentoring in later cohorts;
  • Update mentor list and curriculum as the alignment field’s needs change.

Types of organizations that conduct alignment research, differentiated by funding model and associated market forces:

• Academic research groups (e.g., Krueger's lab at Cambridge, UC Berkeley CHAI, NYU ARG, MIT AAG);
• Research nonprofits (e.g., ARC Theory, MIRI, FAR AI, Redwood Research);
• "Mixed funding model" organizations:
  • "Alignment-as-a-service" organizations, where the product directly contributes to alignment (e.g., Apollo Research, Aligned AI, ARC Evals, Leap Labs);
  • "Alignment-on-the-side" organizations, where product revenue helps funds alignment research (e.g., Conjecture);
• Scaling labs, where alignment research is mostly directed towards improving product (e.g., Anthropic, DeepMind, OpenAI).

The Summer 2023 Cohort has 460 applicants. Our last cohort included 57 scholars.

As an educational seminar and independent research program, MATS cannot offer J1 visas. We can support scholars' ESTA and B1/B2 visa applications, however.

John's scholars have historically only had to seek LTFF funding for the 4-month extension program subsequent to the in-person Scholars Program. They are otherwise treated like other scholars.

Hi Pulkit. Unfortunately, applications have closed for our Summer 2023 Cohort. Hopefully, we will launch applications for our Winter Cohort soon!

I'm somewhere in the middle of the cognitivist/enactivist spectrum. I think that e.g. relaxed adversarial training is motivated by trying to make an AI robust to arbitrary inputs it will receive in the world before it leaves the box. I'm sympathetic to the belief that this is computationally intractable; however, it feels more achievable than altering the world in the way I imagine would be necessary without it.

I'm not an idealist here: I think that some civilizational inadequacies should be addressed (e.g., better cooperation and commitment mechanisms) concurrent with in-the-box alignment strategies. My main hope is that we can build an in-the-box corrigible AGI that allows in-deployment modification.

I agree with you that AI is generally seen as "the big thing" now, and we are very unlikely to be counterfactual in encouraging AI hype. This was a large factor in our recent decision to advertise the Summer 2023 Cohort via a Twitter post and a shout-out on Rob Miles' YouTube and TikTok channels.

However, because we provide a relatively simple opportunity to gain access to mentorship from scientists at scaling labs, we believe that our program might seem attractive to aspiring AI researchers who are not fundamentally directed toward reducing x-risk. We believe that accepting such individuals as scholars is bad because:

• We might counterfactually accelerate their ability to contribute to AI capabilities;
• They might displace an x-risk-motivated scholar.

Therefore, while we intend to expand our advertising approach to capture more out-of-network applicants, we do not currently plan to reduce the selection pressures for x-risk-motivated scholars.

Another crux here is that I believe the field is in a nascent stage where new funders and the public might be swayed by fundamentally bad "AI safety" projects that make AI systems more commercialisable without reducing x-risk. Empowering founders of such projects is not a goal of MATS. After the field has grown a bit larger while maintaining its focus on reducing x-risk, there will hopefully be less "free energy" for naive AI safety projects, and we can afford to be less choosy with scholars.

Mentorship is critical to MATS. We generally haven't accepted mentorless scholars because we believe that mentors' accumulated knowledge is extremely useful for bootstrapping strong, original researchers.

Let me explain my chain of thought better:

1. A first-order failure mode would be "no one downloads experts' models, and we grow a field of naive, overconfident takes." In this scenario, we have maximized exploration at the cost of accumulated knowledge transmission (and probably useful originality, as novices might make the same basic mistakes). We patch this by creating a mechanism by which scholars are selected for their ability to download mentors' models (and encouraged to do so).
2. A second-order failure mode would be "everyone downloads and defers to mentors' models, and we grow a field of paradigm-locked, non-critical takes." In this scenario, we have maximized the exploitation of existing paradigms at the cost of epistemic diversity or critical analysis. We patch this by creating mechanisms for scholars to critically examine their assumptions and debate with peers.