Thanks for responding, and apologies for the belated response in turn. Generally, I agree with all of this, but I think it might miss the point that I had in mind. What I intended to say was that a reward function performs two separate roles that can in principle be separated.
The first is to provide what I elsewhere call a success metric (contrasting it with learned target states). The other is to provide evidence of whether that success metric is achieved or not.
By a constitutive interpretation, I meant that the reward signal itself constitutes the success metric, in which case this distinction is beside the point. However, in principle, one could also define the success metric as a latent variable that a reward signal provides observations of, in which case the distinction applies.
I take all of this to be mostly orthogonal to the question of whether trained agents are aware of their reward function, for example.
Note: I am not an alignment researcher, though I have tried some alignment work before. This text is an attempt to clarify some concepts in a way that I think is important and that might help others too, without spending too much time doing so.
I believe that there are two distinct ways in which the idea of a goal is used in discussions about alignment and agency more broadly. Roughly, the distinction that I have in mind is the following:
I think that there is a further distinction that might be drawn between "constitutive" and "evidential" interpretations of the reward signal.
On the constitutive interpretation, the reward signal is the thing being optimized for, call it intrinsic value. (This is what I take to be the textbook interpretation in RL). On the evidential interpretation, the reward signal is evidence of intrinsic value that the agent uses to update its expected value representations. On this view, the reward signal is more like a perceptual signal but with intrinsic value as its content.
Assuming the evidential interpretation, standard RL models like TD learning are a special case where the reward signal is perfectly accurate, and known... (read more)
I basically agree with the thrust of this post, namely that we need a distinction between our values and goodness. Otherwise, we would not be able to ask the question whether we want what is good, for example. Or to put it differently, there is a conceptual distinction between what is desired and what is desirable, whatever determines the latter.
Furthermore, I agree that it is rather common to see what is desirable as some kind of function of what we in fact value. For example, in economics it is rather common to identify welfare with preference-satisfaction. However, even those who see such close relations between what we want and what would be... (read 454 more words →)
Thanks for responding again. This would probably benefit from a longer and more precise writeup if there is anything of value to be said, but I think that some of the confusion you raised here is something I can clarify.
You are correct that by "success metric" I meant success from the agent's (in this case an AI's) own perspective, not that of a principal aiming to align it. Really, all I had in mind was a framework-neutral expression for the value that is being maximized in any expected value representation of an agent. So this is meant to denote the number that "counts as success" for the agent themselves.
On the "interpretation" point,... (read 374 more words →)