thomascolthurst

(Originally posted at https://plus.google.com/+ThomasColthurst/posts/5fcw5wgVrpj in 2013) I love the premise of Peter de Blanc's "Ontological Crises in Artificial Agents’ Value Systems": you are robot who used to think the universe was described by some set of states O1, only to wake up and find out that the universe is better described by some set of states O2. What should you do? Specifically, let's say that the robot used to act so as to maximize a utility function U: O1 -> R. The problem is to construct a new utility function U': O2 -> R. Briefly, here's the solution Peter de Blanc gives in the paper: assume you have some side information about how probability distributions over O1 and O2 evolve when the robot takes different actions. Use that information to construct a mapping phi: O2 -> O1, and then make U' by composing phi and U: U'(x) = U(phi(x)). I have two quibbles with that approach, one big and one small. The small quibble is that I think it makes more sense to treat phi as a given than as something you need to construct. If we think of the robot as a scientist doing experiments and such, then almost any evidence that the robot gathers that would cause it to change its ontology to O2 would also strongly suggest a mapping from O1 to O2 at the same time. Think of the special theory of relativity, for instance: it didn't just change the space of relative velocities from R to [-c, c], it also described how to compose velocities in the new space, and how to turn accelerations into new velocities. (See http://math.ucr.edu/home/baez/physics/Relativity/SR/rocket.html for an example of the later.) The big quibble is that while U'(x) = U(phi(x)) "type-checks", it probably isn't the utility function the robot really wants. Think of a simple example where O1 is [0,9], O2 is [0,99], phi(x) = floor(x/10), and U(x) = x. Basically, your universe is a line, and you want to get to one end of it, but now you can see more clearly that there are ten times a