You got the basic idea across, which is a big deal.


Though whether it's A or B isn't clear:


A) "this isn't all of the utility function, but its everything that's relevant to making decisions about this right now". ^U doesn't have to be U, or even a good approximation in every situation - just (good enough) in the situations we use it.

Building a building? A desire for things to not fall on people's heads becomes relevant (and knowledge of how to do that).

Writing a program that writes programs? It'd be nice if it didn't produce malware.

Both desires usually exist - and usually aren't relevant. Models of utility for most situations won't include them.


B) The cost of computing the utility function more exactly in the case exceeds the (expected) gains.

isn't clear.

ozziegooen's Shortform

by ozziegooen 31st Aug 2019127 comments