Both 1. "mimic the human" and 2. "maximize according to the human's ontology" will only work well if the human can actually develop a world model (and in the case of 1 also plans) as well as the AI can. If you can do this, then we are probably set on value learning (at least at the level of detail in this post). Moreover, if we can produce world models as good as the AI then we can probably also produce plans as good as the AI, so probably we can just focus on [1]. I'm obviously much more optimistic about this than about approach [3].
Note: I think that the only reason to be interested in approval-directed agents rather than straightforward imitation learners is that it may be harder to effectively imitate behavior than to solve the same task in a very different way. So it seems wrong to say that imitation is most useful as an input into approval-directed agents.
In this post, it seems like ontology identification is acting as a partial solution to the easy goal inference problem, by explaining how the human's behavior would change if their model of the world changed. I would be more interested in seeing any credible strategy for the easy goal inference problem. I don't really see how we are going to solve that problem in a way that doesn't incidentally address ontology identification, and I don't really see how a solution to ontology identification is going to help with that problem.
This may reflect a more general methodological difference. My impression is that the MIRI view is something like "obviously any solution will have to address X, so X is a reasonable way to approach the big problem," for X = ontology identification, tiling, "realistic world models," etc.
My impression is more like "obviously any solution will have to address the ontology identification problem, so it doesn't matter whether we find some special-case solution to that."
Both situations come up a lot in computer science, and I do both of them a ton in my own research. I don't really have a clean story about when one or the other is a more appropriate response, but in these particular cases I have pretty strong intuitions.
Regarding the methodological difference. My perspective is that head-on attempts to solve AI safety are not very promising since we lack the tools to answer basic questions in the general area of the problem such as "what is intelligence?", "what is the computational resource cost of constructing an agent of given intelligence?" or "what is the growth curve of self-improving agents?" Therefore, what we should be doing is constructing a general theory capable of answering questions of this type (I would call it abstract intelligence theory). Thinking about problems such naturalized induction and Vingean reflection seems to me a useful way to approach this, not because they are subproblems of the AI safety problem but because they are handles to getting a mathematical grip on the entire area.
Good point. In this post I was trying to solve something like the easy goal inference problem by factoring it into some different problems, including ontology identification, but it's not clear whether this is the right factoring.
It seems like your intuition is something like: "any way of correctly modelling human mistakes when the human and AI share an ontology will also correctly handle mistakes arising from the human and AI having a different ontology". I think I mostly agree with this intuition. My motivation for working on ontology identification despite this intuition is some combination of (a) "easy" versions of ontology identification seem useful outside the domain of value learning (e.g. making a genie for concrete physical tasks), and (b) I don't see many promising approaches for directly attacking the easy goal inference problem.
But after writing this, I think I have updated towards looking for approaches to the easy goal inference problem that avoid ontology identification. The most promising thing I can think of right now seems to be some variation on planning algorithm 2, but with an adjustment so that the planning can take into account the AI's different predictions (but not the AI's internal representation). It does seem plausible that something in this space would work without directly solving the ontology identification problem.
I don’t see many promising approaches for directly attacking the easy goal inference problem.
I would agree with that. But I don't see how this situation will change no matter what you learn about ontology identification. It looks to me like the easy goal inference problem is probably just ill-posed/incoherent, and we should avoid any approach that rests on us solving it. The kind of insight that would be required to change this view looks very unlike the kind required to solve ontology identification, and also unlike the kind required to make conventional progress in AI, so to the extent that there are workable approaches to the easy goal inference problem it seems like we could work towards them now. If we can't see how to attack the problem, then by the same token that leads me to be pessimistic.
On that perspective we might ask: how we are avoiding the problem? The dodges I know would also dodge ontology identification, by cashing everything out in terms of human behavior. It's harder for me to know what the situation is like for solutions to the goal inference problem--because I don't yet see any plausible solution strategies--but I would guess that the situation will turn out to be similar.
If you concede that you need some kind of "multi-level" model of the world to capture human beliefs about their environment, and in particular if you think that this is necessary for value learning, it seems like you must agree that the game doesn't stop there. Much human knowledge can't be simply described as facts about the world at any level of coarse-graining, at least not in any stronger sense than facts about my dog are facts about the underlying physical data.
That is, facts about my dog can definitely be cashed out as logical facts about the relationship between the underlying physical data + the laws of physics. But they are definitely not explicitly represented as such or conveniently understood as such.
It may be that coarse graining is literally the only way that complex beliefs of this kind work. I would find that surprising in the extreme.
Is anyone defending a position like this, or is the view more something like "well, we know that this is at least one thing that humans do, so we will either (a) address this and then address the next thing and so on, or (b) learn something important about the representation of beliefs/etc. in the course of understanding multi-level models"? Or something very different?
It seems to me like the game probably doesn't stop anywhere sane, so the only option is really for it to stop immediately (probably before you even assume the human is making non-trivial ontological assumptions).
Yeah, I think just having coarse-grained facts would not be enough. I'm referring to a more general idea when I say "multi-level models": something that can represent concepts at different levels of abstraction, probably not with the high-level facts being a function of the low-level facts. My goal would be to have at least some concrete model for a multi-level model that, for example, preserves the "diamond" concept as it learns new physics. I think (a) and (b) are both reasons why I want to do this; I want to know if it's possible to create an AI with goals related to concrete physical things (which requires something like multi-level models, but maybe not much more?), and I also want to have a better understanding of more abstract concepts to see if it's possible to have an AI do anything useful with them.
Could our disagreement be stated as: I think it is plausible that, with a few years of work, a small number of researchers could make useful models for things like diamond-maximization; whereas you don't think it is plausible?
Regarding acting under uncertainty, there is another natural approach namely applying the Nash bargaining solution to an imagery collection of agents with utility functions sampled from the given ensemble.
It's not really clear to me what one would want out of "more explicit models of this instrumental preference for autonomy." It's a complex and messy preference that is tied up with other similarly complex and messy preferences. It probably doesn't have a simple or natural definition in any reasonable ontology.
What concrete questions about this preference would you hope to answer?
To the extent this preference causes a system to have good behavior, it will be because it affects humans' behavior, e.g. a human would predictably and systematically decline actions that significantly reduce their own autonomy. So we need to set up our system so that these effects on human behavior lead to it also avoiding actions that significantly reduce the user's autonomy.
You seem to have in mind a particular version, where the agent infers some latent structure which can then be used to correctly evaluate situations unlike those that appear in the training data (and in particular, to compare the plans put forth by an AI rather than those put forth by a human). So maybe you want to know something about what kind of concepts can robustly transfer from one domain to another quite different domain. It feels to me like you are only going to find bad news here, unless you first make some significant conceptual contributions in AI. So it seems like the first step would be to look for some good news anywhere and see what kind of good news you have to work with. (Or to wait until the AI community produces some good news on its own, and work on other problems in the meantime.)
A somewhat different angle:
The "instrumental goal pursuer" is no more or less dynamically inconsistent than the human. The human wouldn't lock the current goal in place, and so obviously any preferences that successfully explain the human's short-term behavior won't lock the current goal in place. This is a simple observation that already appears in the training set.
This doesn't require learning a complex concept of autonomy. It just requires learning a model of human preferences that roughly reproduces human behavior. If you don't get this kind of thing right, then it seems pretty clear that you aren't going to get useful behavior out of the system in general. Now you may take this as a general argument against value learning, or that value learning will be difficult, but it doesn't seem like we should consider these kinds of preferences as any different from normal preferences.
A model I might want to make would be something like a hierarchical planning algorithm. It would have some supergoal, and then find subgoals of the supergoal. If the system just naively optimized for the subgoals, then it might do silly things like lock its subgoals in place. Instead, this algorithm should prefer plans that maximize the agent's autonomy (in case the agent changes subgoals). If this model works, then maybe we can use it to derive a partial solution to the hard problem of corrigibility. So the real question I want to answer is something like "what kind of AI would a agent with a preference for autonomy choose to build"; I suspect that this AI design will be corrigible in some way. I think this is even useful if the agent is much simpler than a human.
You seem to have in mind a particular version, where the agent infers some latent structure which can then be used to correctly evaluate situations unlike those that appear in the training data (and in particular, to compare the plans put forth by an AI rather than those put forth by a human). So maybe you want to know something about what kind of concepts can robustly transfer from one domain to another quite different domain.
Yeah, this seems accurate. I think this goes back to you being slightly more pessimistic than me about making progress on ontology identification (though I'm still somewhat pessimistic).
This doesn’t require learning a complex concept of autonomy. It just requires learning a model of human preferences that roughly reproduces human behavior.
Right, a good supervised learner should learn this. This is more of a problem if we're using the model's internal representation, not just its predictions.
This is more of a problem if we're using the model's internal representation, not just its predictions.
But you aren't directly using the model's internal representation, are you? You are using it only to make predictions about the human's preferences in some novel domain (e.g. over the consequences of novel kinds of plans).
It seems like it would be cleaner to discuss the whole thing in the context of transfer to a new domain, rather than talking about directly using the learned representation, unless I am missing some advantage of this framing.
Are you hoping to do transfer learning for human preferences in a way that depends on having a detailed understanding of those preferences (e.g. that depends in particular on a detailed understanding of the human preference for autonomy)? I would be very surprised by that. It seems like if you succeed you must be able to robustly transfer lots of human judgments to unfamiliar situations. And for that kind of solution, it's not clear how an understanding of particular aspects of human preferences really helps.
It seems like it would be cleaner to discuss the whole thing in the context of transfer to a new domain, rather than talking about directly using the learned representation, unless I am missing some advantage of this framing.
I agree with this. Problems with learning this preference should cause the system to make bad predictions (I think I was confused when I wrote that this problem only shows up with internal representations). Now that I think about it, it seems like you're right that a system that correctly learns abstract human preferences would also learn the preference for autonomy. So this is really a special case of zero-shot transfer learning of abstract preferences. My main motivation for specifically studying the preference for autonomy is that maybe you can turn a simple version of it into a model for corrigibility.
Are you hoping to do transfer learning for human preferences in a way that depends on having a detailed understanding of those preferences (e.g. that depends in particular on a detailed understanding of the human preference for autonomy)?
I think I mostly want some story for why the preference for autonomy is even in the model's hypothesis space. It seems that if we're already confident that the system can learn abstract preferences, then we could also be confident that the system can learn the preference for autonomy; but maybe it's more of a problem if we aren't confident of this (e.g. the system is only supposed to learn and optimize for fairly concrete preferences).
I don't think the difference is pessimism about ontology identification per se. You overall approach (if successful) seems like it would do zero-shot transfer learning. My perspective would be something like: OK, let's try and understand when we can do zero-shot transfer learning at all, and what assumptions we need to rely on (incidentally, I am also pessimistic about this). You are instead focusing on a different simplification of the problem, one which (I feel) is less likely to be connected to the most important underlying difficulties, and less likely to quickly provide information about whether the overall approach can work.
In the previous post, I discussed three preference frameworks for goal-directed agents. In this post, I will discuss the value-learning sovereign in more detail.
From the Arbital article on genies:
Eliezer Yudkowsky has suggested that people only confront many important problems in value alignment when they are thinking about Sovereigns, but that at the same time, Sovereigns may be impossibly hard in practice. Yudkowsky advocates that people think about Sovereigns first and list out all the associated issues before stepping down their thinking to Genies, because thinking about Genies may result in premature pruning, while thinking about Sovereigns is more likely to generate a complete list of problems that can then be checked against particular Genie approaches to see if they have become any easier.
To this end, I think it is quite useful to discuss how to create a value-learning sovereign, even if it is not a good idea to actually create one. I should be explicit about the fact that the concrete models in this post are almost certainly wrong (even conditioning on the fact that we have to build a value-learning sovereign); they're meant to represent the best concrete illustration of value learning that I can currently write down.
We want the AI to learn human values from human behavior. Usually, values are represented as a utility function. If the type of the world history is , then a utility function over is of type . To learn , we must first have some in mind -- but what could this be? There are 2 plausible candidates:
How might we more formally represent the ontology? A simple environment model for talking about ontologies is the partially observable Markov decision process (POMDP). A POMDP consists of a number of iterations. In each iteration, the agent first takes an action (which causes the state to change stochastically), and then receives an observation of the next state. First we must define the set of actions and observations . These sets apply both to the human and AI. Unlike in a standard POMDP, here the agent's utility function is over the world history rather than the observed reward.
Now let's formally define an ontology. An ontology consists of:
By abuse of notation, let stand both for the type of world histories (list of values), and the ontology itself. Note that this model is Cartesian in much the same way as AIXI is, and therefore faces similar problems. See the paper on realistic world models for more details. It is also unrealistic in that it has no explicit "multi-level" structure; we would expect human and AI concepts to have something like this. The analysis in the rest of the post will be limited by these problems, but I think it will still be useful to analyze an incorrect concrete model.
Each stochastic function in the ontology could be represented by a probabilistic program. For example, consider the following ontology, modelled after the vacuum cleaner example in Artificial Intelligence: A Modern Approach:
# action set A = ['left', 'right', 'suck']
# observation set O = ['clean', 'dirty']
# S consists of (vacuum cleaner location, cleanliness) tuples
# where location is between 0 and 9, and cleanliness is a list of 10 booleans
# indicating whether each square is clean.
def s0():
# start at a random location. Each room is clean with 50% probability.
return (random.randrange(10), [random.random() < 0.5 for i in range(10)])
def st(s, a):
loc = s[0]
cleanliness = s[1][:]
if a == 'left':
# move left
loc = max(0, loc - 1)
if a == 'right':
# move right
loc = min(9, loc + 1)
if a == 'suck':
# probably suck dirt from current square
if random.random() < 0.9:
cleanliness[loc] = True
return (loc, cleanliness)
def o(s):
# observe cleanliness of current square
if s[1][s[0]]:
return 'clean'
else:
return 'dirty'
With the ontology in place, we can consider some utility function over it (in this case, discounted cleanliness over time):
def U(state_seq):
util = 0
discount = 1.0
for s in state_seq:
clean = 0
for c in s[1]:
if c:
clean += 1
util += discount * clean
discount *= 0.99
Since discounts exponentially, it can easily be extended to a utility function over infinite state sequences.
If the AI already has some ontology and some utility function over the ontology , then it is possible for it to search for utility-maximizing policies. A policy could be represented as a stochastic function , which specifies what action the agent takes given an action/observation sequence. Essentially, a policy is a stochastic infinite decision tree. could be chosen to maximize , where is the state sequence and the expectation is with respect to the distribution defined by the ontology .
With the ontology machinery in place, we have the tools to look at a couple proposals for how to learn the human's values.
We assume that the AI has access to the observations and actions of the human.
For example, in the vacuum cleaner example, perhaps the AI sees the sequence
['suck', 'clean', 'right', 'clean', 'right', 'dirty', 'suck', 'clean'],
meaning that the human controlling the vacuum cleaner first decided to suck dirt,
observed that the square was now clean, went right, observed that this square
was also clean, etc. From a long enough sequence like this, the AI should
approximately learn the human's values.
In practice, it might be necessary for the AI to locate humans in the environment. This adds some additional complexity, but for now I will ignore this.
The AI could assume that the human's values are expressed over the (unknown) human ontology . The AI has a joint prior over and a utility function . Additionally, the AI needs to predict how the human behaves given their ontology and utility function. One approach, common in economics, is to assume that the human maximizes expected utility. However, this model is quite unrealistic, and alternative models have been explored in the field of cognitive science.
We could represent an alternative behavioral model as a policy , similar to a planning model. The behavioral model should depend on and . While it is possible to set the behavioral model to maximize expected utility, this is psychologically unrealistic, so the behavioral model should allow the human to sometimes take suboptimal actions.
Suppose the AI has inferred the human ontology , utility function , and behavior model. How could it make plans? There are 3 immediate candidates for planning algorithms:
Planning algorithm 1 is not too interesting by itself; it is probably more useful as an input to other AI control methods, such as approval-directed agents.
Planning algorithm 2 is more interesting. It selects a plan that looks good according to the human's ontology. This does not take advantage of the AI's ability to make better predictions than the human, but it does take advantage of the AI's better ability to search for plans. For example, if I were trying to solve a boolean satisfiability problem, an AI using this algorithm could suggest a solution, because my ontology predicts that this solution works, even though I can't find the solution myself. In this way, an agent using this planning algorithm is similar to an approval-directed agent. The main difference is that it selects a plan (i.e. an infinite stochastic decision tree) that maximizes how good the human expects the results to be, rather than an action. Otherwise, it is quite similar.
Planning algorithm 3 uses the full power of the AI, including the AI's ability to make better predictions than the human. It requires deriving a utility function from the inferred human utility function . If we have , and we want to create , one way to get this is to create a probabilistic ontology mapping function , and then define . The ontology mapping function is meant to say which histories in best represent some history in . Probably, it is more intuitive to map states rather than world histories, but it shouldn't matter much. The paper Ontological Crises in Artificial Agents' Value Systems discusses an ontology mapping method.
While it would be interesting to look more closely at the relation between planning algorithm 2 and approval-directed agents at some point, I'll focus on planning algorithm 3 for the rest of the post. Planning algorithm 3 has multiple problems:
As an alternative to learning the human's ontology and utility function expressed in this ontology, the AI could assume that the human's values are expressed over the (known) AI ontology . The AI has a joint prior over the utility function and the behavior model as before. The assumption is that the human plans using the AI's ontology, rather than a different human ontology. Current value-learning algorithms, such as inverse reinforcement learning and inverse planning, work this way because they do not distinguish between the AI's ontology and the human's ontology.
Unfortunately, this model is psychologically implausible. We do not think of the human's preferences about the AI's ontology being a cause of human behavior; rather, it is the human's preferences about the human's ontology that is a cause of human behavior. One place where this shows up is when the human takes an action that would be irrational if the human were using the AI's ontology (for example, the human calculates something incorrectly because they do not know about quantum physics, which the AI knows about). The AI has no choice but to either believe that the human's utility function considers making this incorrect calculation to be optimal, or to explain it as an error according to the behavior model. For the second option to produce reasonable results, the behavior model must be quite complex; it will probably talk about the human's ontology and how the human's goals in this ontology relate to , much like proposal 1.
Overally, I do not find this proposal promising. Either the behavioral model suffices to explain correlated human errors due to the human having an incorrect ontology (in which case the behavioral model contains all the complexity of proposal 1), or it does not (in which case the AI will learn the wrong values). Therefore, I will talk about the first proposal in the rest of this post.
Paul Christiano has previously written about the distinction between learning terminal and instrumental goals in his post, Ambitious vs. narrow value learning. It is possible to explore this distinction in proposal 1. Since the human’s utility function is relative to the human’s ontology, it is not possible for it to express truly terminal goals. Determining humans’ reflective preferences about states of the universe requires some kind of philosophical extrapolation process, in which humans clarify their concepts and develop preferences about their new concepts.
However, by varying the behavioral model, it is possible to learn either higher-level instrumental goals (for example, getting a job), or lower-level instrumental goals (for example, filling out a particular job application). If the behavior model states that the human behaves by finding subgoals of and then optimizing for them (as we would expect if were a high-level goal), then it is more likely to detect high-level goals. On the other hand, if the behavior model states that the human optimizes for more directly (as we would expect a human to optimize for a low-level goal), then it is more likely to detect low-level goals.
Note that, since instrumental goals change over time, we would also need to have change over time. This is a simple modification to the original model. Obviously, the AI’s goal should be set so it has no incentive to change the human’s goals to make them easier to optimize. Perhaps its goal at time is to maximize expected utility of whatever is at time .
Naively, if the AI’s utility function changes over time, then it will be dynamically inconsistent. The AI at an earlier time will have an incentive to lock the current value of in place, so that future versions of the AI will optimize for this instead of whatever is estimated to be in the future. This would lead to a system that determines what my instrumental preferences are, and then continues to optimize for these even as my instrumental preferences change.
It seems that a system that locks my "object-level" instrumental goals (such as filling out a job application) in place would be acting against some of my other instrumental goals: specifically, my instrumental goal of preserving my autonomy. Paul discusses this preference in his post:
Humans have many clear instrumental goals like "remaining in effective control of the AI systems I deploy," "acquiring resources and other influence in the world," or "better understanding the world and what I want." A value learner may able to learn robust preferences like these and pursue those instrumental goals using all of its ingenuity.
In general, I will prefer plans that maximize my autonomy, so I could consider autonomy-maximization to be one of my instrumental goals. This preference could explain my desire to study moral philosophy, even when this might cause my moral opinions to change (and therefore be bad according to my current object-level moral views). By caring about my autonomy, I can mostly preserve dynamic consistency even as my goals change over time.
More concretely, suppose the state in the human's ontology contains a field for "autonomy", indicating how much autonomy I have in this state. We would hope that state sequences in in which the human has low autonomy get mapped to sequences of states in that have a low autonomy field. For example, state sequences in which the AI manipulates the human should be mapped to states with a low autonomy field.
Of course, it would be imprudent to assume that proposal 1 will correctly do all this. "Human autonomy" seems to be a complex concept, so it would be difficult to learn. To reduce confusion, it would be a good idea to create more explicit models of this instrumental preference for autonomy. This seems related to the hard problem of corrigibility: the human's desire for AIs to be corrigible is really a reflection of the human's preference for autonomy. This seems somewhat related to hierarchical planning, so maybe I will have better models of this preference after understanding hierarchical planning better.
If a model like this works, then we can ground human values in something other than terminal goals: specifically, systems of instrumental goals at each time step that chain together in a tiling fashion, with each instrumental goal system trusting the next under normal circumstances. I think this is a promising alternative way to look at human values, though I still lack concrete models for this.
The system should have uncertainty about the correct values. In both proposals, the human utility function is underdetermined by the data. In proposal 1, the human ontology is underdetermined by the data, and additionally any uncertainty about the correct ontology mapping method propagates into uncertainty about the correct utility function.
Under uncertainty about the correct utility function, it is not straightforward to simply maximize expected utility. This is because the "loudness" of different possible preferences matters. Given this, there are 2 clear ways to act under uncertainty:
I think it's plausible that some variant of minimax works for conservatively optimizing values under uncertainty, so more research in this area could be useful.
I claim that, for proposal 1 to work, the ontology mapper needs to be very powerful and reliable. This is because:
These problems make me quite pessimistic about this proposal. More research into ontology identification might yield insights about just how hard these problems are to solve.
Suppose the AI has created a plan . Humans could examine this plan by seeing what state sequences result from this plan (assuming humans understand the ontology). There are 2 obvious ways to do this:
I don't think any of the models I have described will do anything useful with superhuman intelligence. The most potentially powerful models require essentially solving cognitive science (to get the behavioral model), and creating an overpowered ontology mapper. Still, I have identified concrete areas for further research, which might turn up results useful for both value-learning sovereigns and other agents. Specifically, further research into value-learning sovereigns could look at: