epistemic status: Working notes of three different people on the same question, likely useless/incomprehensible to anyone else

The question

How to find the right abstraction level of human values

Problems in over- or underfitting human values:

We can learn human values by observing their actions and distilling them into a preference relation. This learned preference relation can overfit human values (eg: Humans want to raise their left arm by 2 cm on 2022-05-07 if they’re in some specific place) or it can underfit human values (eg: Humans care only about maximizing money). If our preference relation overfits, we expect to not find some known biases, e.g. the Allais Paradox. There are also both inconsistencies that are “too abstract” and “too concrete”:

Too abstract: If I have three cities $B$ , $H$ , and $M$ , and I traveled $B \to H, H \to M, M \to H$ , then one might conclude that I have an inconsistency, but in reality I made the travels $M_{1} \to B_{1}, B_{1} \to H_{1}$ and $H_{2} \to M_{2}$
Too concrete: (?) If I hugged my friend at location $l_{1}$ and time $t_{1} (H_{11})$ , but not at $l_{1}$ and $t_{2} (H_{12})$ , but the information about time was disregarded, we might conclude that $H_{1} \leftrightarrow H_{1}$ , which is inconsistent, but in reality I pretty much always want to hug my friend, regardless of time and place.

For a set of worlds W, the learned preference relation $P \subseteq W \times W$ represents which world is preferred to another (this generates a graph $G = (W, W \times W)$ which can be any graph, with cycles & disconnected components). If we overfit human values, we assume they’re way more rational than they actually are, if we underfit them, we assume they’re way less rational than they actually are. So there is a spectrum over the complexity of the learned preference relation: from overfitting/complexity/rationality/concreteness to underfitting/simplicity/irrationality/abstraction.

Cognitive biases as a lower bound for finding the right abstraction level of human preferences

Behavior commonly considered irrational can give pointers towards which abstraction level to use. Literature on cognitive biases give a lower bound on the abstraction level of human values. For example, scope neglect is only a bias if we consider the preference “Save the most birds possible” to be our actual preference, and not “Get the most pleasure possible out of saving birds for the least cost of saving them”. A learner having inferred the first preference will judge humans as exhibiting ...

Posts

Wikitag Contributions

Comments