His point is that if I fix your goals (say, narrow self-interest) the defensible policies still don't look much like short-sighted goal pursuit (in some environments, for some defensible notions of "defensible"). It may be that all sufficiently wise agents pursue the same goals because of decision theoretic considerations, by implicitly bargaining with each other and together pursuing some mixture of all of their values. Perhaps if you were wiser, you too would pursue this "overgoal," and in return your self-interest would be served by other agents in the mixture.
While plausible, this doesn't look super likely right now. Will would get a few Bayes points if it pans out, though the idea isn't due to him. (A continuum of degrees of altruism have been conjectured to be justified from a self-interested perspective, if you are sufficiently wise. This is the most extreme, Drescher has proposed a narrower view which still captures many intuitions about morality, and weaker forms that still capture at least a few important moral intuitions, like cooperation on PD, seem well supported.)
The connection to omega isn't so clear. It looks like it could just be concealing some basic intuitions about computability and approximation. It seems like a way of smuggling in mysticism, which is misleading by being superfluous rather than incoherent.
The connection to omega isn't so clear. It looks like it could just be concealing some basic intuitions about computability and approximation. It seems like a way of smuggling in mysticism, which is misleading by being superfluous rather than incoherent.
I thought about it some more and remembered one connection. I'll post it to the discussion section if it makes sense upon reflection. The basic idea is that Agent X can manipulate the prior of Agent Y but not its preferences, so Agent X gives Agent Y a perverse prior that forces it to optimize for the preferences of Agent X. Running this in reverse gives us a notion of an objectively false preference.
Here's the new thread for posting quotes, with the usual rules: