All of Khoth_duplicate0.7536578752539433's Comments + Replies

I think there are two ways that a reward function can be applicable:

1) For making moral judgements about how you should treat your agent. Probably irrelevant for your button presser unless you're a panpsychist.

2) If the way your agent works is by predicting the consequences of its actions and attempting to pick an action that maximises some reward (eg a chess computer trying to maximise its board valuation function). Your agent H as described doesn't work this way, although as you note there are agents which do act this way and produce the same behaviour ... (read more)

My "solution" does use 2), and should be posted in the next few days (maybe on lesswrong 2 only - are you on that?)