Decision Theory

abramdemski; Scott Garrabrant

(A longer text-based version of this post is also available on MIRI's blog here, and the bibliography for the whole sequence can be found here.)

I don't quite follow why 5/10 example presents a problem.

Conditionals with false antecedents seem nonsensical from the perspective of natural language, but why is this a problem for the formal agent? Since the algorithm as presented doesn't actually try to maximize utility, everything seems to be alright. In particular, there are 4 valid assignments: , $p_{2} : (x = 5, y = 0, A () = 5, U () = 5)$ , $p_{3} : (x = 5, y = 10, A () = 10, U () = 10)$ , $p_{4} : (x = 10, y = 10, A () = 10, U () = 10)$

The algorithm doesn't try to select an assignment with largest $U ()$ , but rather just outputs $5$ if there's a valid assignment with $x > y$ , and $10$ otherwise. Only $p_{2}$ fulfills the condition, so it outputs $5$ . $p_{1}$ and $p_{4}$ also seem nonsensical because of false antecedents but with attached utility $U () = 10$ - would that be a problem too?

For this particular problem, you could get rid of assignments with nonsensical values by also considering an algorithm with reversed outputs and then taking the intersection of valid assignments, since only $(x = 5, y = 10)$ satisfies both algorithms.

2abramdemski5y

Yep, agreed. I used the language "false antecedents" mainly because I was copying the language in the comment I replied to, but I really had in mind "demonstrably false antecedents".

4Ian Televan5y

While I agree that the algorithm might output 5, I don't share the intuition that it's something that wasn't 'supposed' to happen, so I'm not sure what problem it was meant to demonstrate. I thought of a few ways to interpret it, but I'm not sure which one, if any, was the intended interpretation: a) The algorithm is defined to compute argmax, but it doesn't output argmax because of false antecedents. - but I would say that it's not actually defined to compute argmax, therefore the fact that it doesn't output argmax is not a problem. b) Regardless of the output, the algorithm uses reasoning from false antecedents, which seems nonsensical from the perspective of someone who uses intuitive conditionals, which impedes its reasoning. - it may indeed seem nonsensical, but if 'seeming nonsensical' doesn't actually impede its ability to select actions wich highest utility (when it's actually defined to compute argmax), then I would say that it's also not a problem. Furthermore, wouldn't MUDT be perfectly satisfied with the tuplep1:(x=0,y=10,A()=10,U()=10) ? It also uses 'nonsensical' reasoning 'A()=5 => U()=0' but still outputs action with highest utility. c) Even when the use of false antecedents doesn't impede its reasoning, the way it arrives at its conclusions is counterintuitive to humans, which means that we're more likely to make a catastrophic mistake when reasoning about how the agent reasons. - Maybe? I don't have access to other people's intuitions, but when I read the example, I didn't have any intuitive feeling of what the algorithm would do, so instead I just calculated all assignments (x,y)∈{0,5,10}2, eliminated all inconsistent ones and proceeded from there. And this issue wouldn't be unique to false antecedents, there are other perfectly valid pieces of logic that might nonetheless seem counterintuitive to humans, for example the puzzle with islanders and blue eyes. ---------------------------------------- [...] When I try to examine my own reaso

While I agree that the algorithm might output 5, I don't share the intuition that it's something that wasn't 'supposed' to happen, so I'm not sure what problem it was meant to demonstrate.

OK, this makes sense to me. Instead of your (A) and (B), I would offer the following two useful interpretations:

1: From a design perspective, the algorithm chooses 5 when 10 is better. I'm not saying it has "computed argmax incorrectly" (as in your A); an agent design isn't supposed to compute argmax (argmax would be insufficient to solve this problem, because we're not g... (read more)

136

Decision Theory

136

Ω 34

136

Ω 34

136

Decision Theory

136

Ω 34

136

Ω 34