Epistemic status: I'm describing a problem I have, which probably has an existing solution.
1.
Let's say you're reading an article, and there's one sentence in particular that doesn't stick out to you. The sentence seems intuitive, so your eyes slide over it and continue down the page.
This happens when your intuition matches with the sentence. The sentence follows some subconscious prediction. It does not surprise you. You extract no evidence from the sentence, and you don't update your beliefs.
The sentence is a missed opportunity. Its presence narrows the possibilities for the territory (unless you really did predict those words in that order). Some update should happen.
The immediate territory here is, which words the... (read 430 more words →)
I know the answer to "couldn't you just-" is always "no", but couldn't you just make an AI that doesn't try very hard? i.e., it seeks the smallest possible intervention that ensures 95% chance of whatever goal it's intended for.
This isn't a utility maximizer, because it cares about intermediate states. Some of the coherence theorems wouldn't apply.