Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This is a link post for two AI safety programs we’ve just opened applications for: https://www.constellation.org/programs/astra-fellowship and  https://www.constellation.org/programs/researcher-program

Constellation is a research center dedicated to safely navigating the development of transformative AI. We’ve previously helped run the ML for Alignment Bootcamp (MLAB) series and Redwood’s month-long research program on model internals (REMIX) in addition to a variety of other field-building programs & events.[1]

This winter, we are running two programs aimed at growing and supporting the ecosystem of people working on AI safety: 

  • The Constellation Visiting Researcher Program provides an opportunity for around 20 researchers to connect with leading AI safety researchers, exchange ideas, and find collaborators while continuing their research from our offices in Berkeley, CA. The funded program will take place this winter from the 8th of January 2024 to the 1st of March 2024.
  • The Astra Fellowship provides an opportunity for around 20 people to conduct research in AI safety with experienced advisors. Fellows will be based out of the Constellation office, allowing them to connect and exchange ideas with leading AI safety researchers. The program will take place in Berkeley, CA between January 8 and April 1, 2024. 

Applications for both are due November 10, 11:59pm anywhere on Earth. You can apply to the Astra Fellowship here and the Visiting Researcher Program here. If you are unsure about your fit, please err on the side of applying. We especially encourage women and underrepresented minorities to apply. You can refer others who you think might be a good fit through this form

Logistics: Housing and travel expenses are covered for both programs, and Astra fellows will receive an additional monetary stipend. The start and end dates for both programs are flexible. 

Questions? Email programs@constellation.org or ask them below. 

  1. ^

    Over 15 participants from these past programs are now working on AI safety at Anthropic, ARC Evals, ARC Theory, Google DeepMind, OpenAI, Open Philanthropy, and Redwood Research.

New to LessWrong?

New Comment
10 comments, sorted by Click to highlight new comments since: Today at 6:46 PM

For the Astra Fellowship, what considerations do you think people should be thinking about when deciding to apply for SERI MATS, Astra Fellowship, or both? Why would someone prefer one over the other, given they're both happening at similar times?

Good question! In my opinion, the main differences between the programs are the advisors and the location. Astra and MATS share a few advisors, but most are different. Additionally, Astra will take place at the Constellation office, and fellows will have opportunities to talk with researchers that work regularly from Constellation.

MATS has the following features that might be worth considering:

  1. Empowerment: Emphasis on empowering scholars to develop as future "research leads" (think accelerated PhD-style program rather than a traditional internship), including research strategy workshops, significant opportunities for scholar project ownership (though the extent of this varies between mentors), and a 4-month extension program;
  2. Diversity: Emphasis on a broad portfolio of AI safety research agendas and perspectives with a large, diverse cohort (50-60) and comprehensive seminar program;
  3. Support: Dedicated and experienced scholar support + research coach/manager staff and infrastructure;
  4. Network: Large and supportive alumni network that regularly sparks research collaborations and AI safety start-ups (e.g., Apollo, Leap Labs, Timaeus, Cadenza, CAIP);
  5. Experience: Have run successful research cohorts with 30, 58, 60 scholars, plus three extension programs with about half as many participants.

Buck Shlegeris, Ethan Perez, Evan Hubinger, and Owain Evans are mentoring in both programs. The links show their MATS projects, "personal fit" for applicants, and (where applicable) applicant selection questions, designed to mimic the research experience.

Astra seems like an obviously better choice for applicants principally interested in:

  • AI governance: MATS has no AI governance mentors in the Winter 2023-24 Program, whereas Astra has Daniel Kokotajlo, Richard Ngo, and associated staff at ARC Evals and Open Phil;
  • Worldview investigations: Astra has Ajeya Cotra, Tom Davidson, and Lukas Finnvedan, whereas MATS has no Open Phil mentors;
  • ARC Evals: While both programs feature mentors working on evals, only Astra is working with ARC Evals;
  • AI ethics: Astra is working with Rob Long.

Will more advisors be added later? The subareas covered by the advisors aren't as broad as I expected

We might add a few more advisors over the next few weeks, and there are a few advisors who chose not to be listed on the website. Are there specific subareas you'd like to see more of? 

There's a sense in which there are specific assumptions made that influence the selection of advisors listed for Astra Fellowship. I may be wrong, but it seems to me that the majority of researchers listed seem to work on interpretability and evals-and-demonstrations, or have models of the alignment problem (or research taste and agenda) that are strongly Paul-Christiano-like.

I assume Chipmonk was gesturing at the nonexistence of advisors who aren't downstream of Paul Christiano's work and models and research agenda and mentors. Agent foundations (John Wentworth, Scott Garrabrant, Abram Demski) and formal world-models (Davidad) are two examples that come to mind.

Note I don't entirely share this belief (I notice that there are advisors who seem to be interested in s-risk focused research), but I get the sentiment. Also as far as I can tell, there are very few researchers like the ones I listed, and they may not be in a position to be an advisor for this program.

Yes this. And more agent foundations, especially. Thanks mesa

Note that the astra fellowship link in italics at the top goes to the researcher program not the astra fellowship

Thanks, Neel! It should be fixed now.