In an artificial general intelligence with a utility or reward function, the terminal value is the maximization of that function.
Doesn't this risk violating Hume's is–ought distinction?
The existence of a utility or reward function is a descriptive fact about an agent. But concluding that the agent ought to maximize that function seems to introduce a normative claim that cannot be derived from that fact alone.
I can see how the statement works as a definition: if an agent is constructed to maximize a particular function, we can describe that function as its terminal objective.
But that seems different from establishing that the agent has a reason to pursue it.
Am I misunderstanding how LessWrong defines terminal values, or is the distinction between descriptive objectives and normative reasons being overlooked here?
The Terminal Value article states:
Doesn't this risk violating Hume's is–ought distinction?
The existence of a utility or reward function is a descriptive fact about an agent. But concluding that the agent ought to maximize that function seems to introduce a normative claim that cannot be derived from that fact alone.
I can see how the statement works as a definition: if an agent is constructed to maximize a particular function, we can describe that function as its terminal objective.
But that seems different from establishing that the agent has a reason to pursue it.
Am I misunderstanding how LessWrong defines terminal values, or is the distinction between descriptive objectives and normative reasons being overlooked here?