[[FYI: I'm just copying this in and removing a few bits; apologies for formatting; I don't intend to post the attachments]]

EtA: I (embarrasingly and unfortunately) understated Richard Ngo's technical background in email; I left what I originally wrote in ~~strikethrough.~~

EtA(2): I thought a bit more context would be useful here:
- This email was designed for a particular person after several in person conversations.
- I'm not proposing this as anything like "The best things to send to an arbitrary 'experienced ML researcher interested in learning about Alignment / x-safety'".
- I didn't put a ton of effort into this.
- I aimed to present a somewhat diverse and representative sampling of x-safety stuff.

EtA (3): Suggestions / Feedback welcome!

OK I figure perfect is the enemy of the good and this is already a doozy of an email, so I'm just going to send it :)

I think it would be great to get a better sense of what sort of materials are a good introduction for someone in your situation, so please let me know what you find most/least useful/etc.!

A few top recommendations...

- read about DeepMind's "safety, robustness, and assurance" breakdown of alignment
- sign up for the Alignment and ML Safety newsletters, and skim through the archives

- Look at this syllabus I mocked up for my UofT application (attached). I tried to focus on ML and include a lot of the most important ML papers, although there's a lot missing.

- Read about RL from human preferences if you haven't already. I imagine you might've seen the "backflipping noodle" blog post. I helped author a research agenda based on such approaches Scalable agent alignment via reward modeling. The research I talked about in my talk is part of a project on understanding reward model hacking; we wrote a short grant proposal for that (attached). I'm in the midst of rethinking how large a role I expect reward modeling (or RL more generally) to play in future AI systems, but I previously considered this one of the highest priority directions to work on. Others (e.g. Jan, first author of the agenda) are working on showing what you can do with reward modeling; I'm more concerned with figuring out if it is a promising approach at all or not (I suspect not, because of power-seeking / instrumental convergence).

- Look at the "7 Alternatives for agent alignment" in the agenda for a brief overview of alte...

icyflame

icyflame