Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

To quote Andrew Critch:

I get a lot of emails from folks with strong math backgrounds (mostly, PhD students in math at top schools) who are looking to transition to working on AI alignment / AI x-risk. There are now too many people “considering” transitioning into this field, and not enough people actually working in it, for me, or most of my colleagues at Stuart Russell’s Center for Human Compatible AI (CHAI), to offer personalized mentorship to everyone who contacts us with these qualifications.

From math grad school to AI alignment, Andrew Critch

I’m pretty sure he wrote that at least 4 years ago (2016 or earlier). The field has grown enormously since then, but so has the number of people considering it as a research area. So far, I’ve tried to give at least 10 minutes of my time to anyone who emails me with questions; that probably won’t be sustainable for much longer. So now I’m answering the questions I get most frequently.  I hope to keep this up to date, but no promises.

Usually, I write a blog post when I think I have something important and novel to say, that I am relatively confident in. That’s not the case for this post. This time, I’m taking all the questions that I frequently get and writing down what I’d say in response. Often, this is (a) not that different from what other people would say, and (b) not something I’m very confident in. Take this with more grains of salt than usual.

Thanks to Neel Nanda, Nandi Schoots, and others who wish to remain anonymous for contributing summaries of conversations.

See the linked post for the FAQ; which will hopefully be kept up to date over time.

New Comment
2 comments, sorted by Click to highlight new comments since: Today at 9:51 PM

I still don’t know much about GANs, Gaussian processes, Bayesian neural nets, and convex optimization.

FWIW, I'd recommend investing the time to learn about convex optimization even if you don't think you need it yet. Unlike the other topics on this list, the ideas from convex optimization are relevant to a wide variety of things the name does not suggest. Some examples:

  • how to think about optimization in high dimensions (even for non-convex functions)
  • using constraints as a more-gearsy way of representing relevant information (as opposed to e.g. just wrapping that information into an objective function directly)
  • quantifying slackness/tautness (which is one of the main benefits of representing information as constraints)
  • useful intuitions for making numerical algorithms efficient, especially leveraging sparsity and recognizing failure modes of poor condition numbers

Personally, I consider convex optimization ideas fundamentally useful for thinking in a similar way to linear algebra or multivariate calculus.

(Note: I first learned the topic mainly from Boyd's lectures; he's unusually good at conveying many useful intuitions in those talks, so you might get less value from other lecturers or the canonical convex optimization book.)

Yeah that's definitely the one on the list that I think would be most useful.

I may also be understating how much I know about it; I've picked up some over time, e.g. linear programming, minimax, some kinds of duality, mirror descent, Newton's method.