I can see how my last comment may have made it seem like I thought some terminal goals should be protected just because they are terminal goals. However, when I said that Gandhi's anti-murder goal and the egoist's self-indulgence goal might have distinct features that not all terminal goals share, I only meant that we need a broad definition of terminal goals to make sure it captures all varieties of terminal goals. I didn't mean to imply anything about the relevance of any potential differences between types of terminal goals. I would not assume that whatever distinguishes an egoist's goal of self-indulgence from an AI's goal of destroying buildings means the egoist... (read more)
Ha, yes, fair enough
"When an entity 'cares about X like ghandi cares about avoiding murder' or 'cares about X like a pure egoist cares about his own pleasure' I would call that 'having X as terminal goal.'"
I think I would agree with this, unless you would also claim that "caring about X like a pure egoist cares about his own pleasure" is the only way of having a terminal goal. I would define a terminal goal more broadly as a non-instrumental goal: a goal pursued for its own sake, not for anything else. How a pure egoist cares about his own pleasure might have particular features that some non-instrumental goals might not have. I would... (read more)
Maybe this is still B, in which case I might have interpreted it more strictly than you intended.
They could be using their current goal to evaluate the future, but include in the future that they won't have that goal. This doesn't require excluding this goal from their analysis all altogether. It's just that they evaluate that the failure of this goal is irrelevant in a future in which they don't have the goal.
It just occurred to me that since you implied that ends-rationality would make goal abandonment less likely, you might be using it in a different way than me, to refer to terminal goals. The paper assumes an AI will have terminal goals, just as humans do, and that these terminal goals are what can be abandoned. Ends-rationality provides one route to abandoning terminal goals. The paper's argument is that goal abandonment is also possible without this route.
The paper argues that the number of failures in 2 (goal abandonment) is also 0. This is because it is no longer her goal once she abandons it. She fails by "the goal" but never fails by "her goal." Cake isn't the best case for this. The argument for this is in 3.4 and 3.5.
The instrumental convergence thesis doesn't depend on being applied to a digital agent. It's supposed to apply to all rational agents. So, for this paper, there's no reason to assume the goal takes the form of code written into a system.
There may be a way to lock an AI agent into a certain pattern of behaviour or a goal that it can't revise, by writing code in the right way. But if an AI keeps its goal because it can't change its goal, that has nothing to do with the instrumental convergence thesis.
If an agent can change its goal through self-modification, the instrumental convergence thesis could be relevant. If an agent could... (read more)
I don't assume A or B. The argument is not about what maximally satisfies an agent. Goal abandonment need not satisfy anything. The point is just that goal abandonment does not dissatisfy anything.
This paper of mine ("A Timing Problem for Instrumental Convergence"), co-authored with Helena Ward and Jen Semler, was recently accepted in Philosophical Studies for a superintelligent robots issue (open access). The paper argues that instrumental rationality doesn't require goal preservation/goal-content integrity/goal stability. Here is the abstract:
... (read more)Those who worry about a superintelligent AI destroying humanity often appeal to the instrumental convergence thesis—the claim that even if we don’t know what a superintelligence’s ultimate goals will be, we can expect it to pursue various instrumental goals which are useful for achieving most ends. In this paper, we argue that one of these proposed goals is mistaken. We argue that instrumental goal preservation—the claim that
"It's plausible that AIs will have self-preserving preferences (e.g. like E[sum_t V_t0(s_t)]). It is likely we will build such AIs because this is roughly how humans are, we don't have a good plan to build very useful AIs that are not like that, and current AIs seem to be a bit like that. And if this is true, and we get V even slightly wrong, a powerful AI might conclude its values are better pursued if it got more power, which means self-preservation and ultimately takeover."
This strikes me as plausible. The paper has a narrow target. It's arguing against the instrumental convergence argument for goal preservation. It argues that we shouldn't expect... (read more)