At the time of writing, I have just (nearly) finished my PhD at Oxford. During that time, most of my main research has been motivated by the goal of developing a theoretical foundation for the field of reward learning. The purpose of this sequence is to explain and motivate this research agenda, and to provide an accessible overview over some of my results. It will be divided into several smaller posts:

First, in this post, I will define and motivate the theoretical reward learning research agenda, including my aims and background assumptions.
Over a series of subsequent posts, I will provide overviews of and comments on some of the existing research within this research agenda.
Finally, in the last post, I will also provide some resources for anyone who wants to contribute to this (or similar) research, in the form of both open problems and some thoughts on how these problems could be approached.

I hope that this sequence will make it easier to get an overview over my recent research. I also want to welcome discussion, so please make a comment if you have any questions. Finally, I think this research agenda is promising, and it has proven to be very tractable, yet not many people work on it (or in related areas). I therefore also want to advertise this research, and make this post a good starting point for people who may be interested in working on similar questions themselves.

Defining the Theoretical Reward Learning Research Agenda

In one sentence, the aim of this research agenda is to develop a mature theoretical foundation for the field of reward learning (and relevant adjacent areas). This is very closely related to work that has been done in the AI safety space for quite some time, but I have approached it from a slightly different angle. Concretely, this research agenda involves answering questions such as:

What is the right method for expressing goals and instructions to AI systems?
1. The current default answer to this question is “exponentially discounted scalar Markovian reward functions." However, this is not the only option, and it is not self-evident that it is the right choice. Some other notable options include e.g. multi-objective RL, temporal logic, or different kinds of non-Markovian rewards.
2. Some relevant criteria for evaluating a specification language include:
  1. How expressive is the language? Are there things it cannot express?
  2. How intuit

...

The Theoretical Foundations of Reward Learning