Crosspost from the EA Forum.

Why think about goalsets?

Societies need many distinct systems: a transport system, a school system, etc. These systems cannot be justified if they are amoral, so they must serve morality. Each system cannot, however, achieve the best moral outcome on its own: If your transport system doesn’t cure cancer, it probably isn’t doing everything you want; if it does cure cancer, it isn’t just a “transport” system. So each system must have a bounded domain and, ideally, still be morally optimal. To solve this problem, systems can be assigned goals, which are statements that we want the system to satisfy. Goals weakly constrain a system’s domain because if a system satisfies its goals, there is no more for it to do.

The problem, then, is deciding which goals to pursue. Each goal should not be evaluated on its own. Instead, we should evaluate your entire set of goals, your goalset. There are two reasons for this. (1) Any problem with a goal will manifest in a goalset that contains it, so we can rephrase any criticisms of a goal in terms of goalsets. And (2) there are times when we have to look at the goalset as a whole before identifying an issue. For example, suppose two identical twins, Robinson and Crusoe, are stranded on an island. Suppose the best outcome occurs when either Robinson hunts for food and Crusoe builds a shelter or vice versa. So “Crusoe builds a shelter” is a goal that fits in one ideal outcome, and “Robinson builds a shelter” fits in another. However, if their goalset contains both those statements, the pair will starve (albeit in the comfort of an excellent shelter).

Clearly, your goalset should be related in some way to your morality. But how, precisely? Can goalsets even be a component of making ideal systems? Does it make sense to talk about “ideal” systems? What does it mean for a system to “satisfy” a goal? What are the “consequences” of a goalset? What is “equivalence” between goalsets? We’ll define all of these terms from the same four objects, which allows us to (1) see how these concepts relate to each other, (2) generate theorems about policy arguments, and (3) provide stronger foundations for informal reasoning.

Some of the later conclusions

Vague goals have no place within a goalset. However, we can satisfy concrete goals that are related to the vague goals.
If goals are “narrow” when they are not decomposa

...