Doesn't this merely meta sidestep the issue? Now what the AI needs to do is modify itself to use pragmatic goals when modifying itself, to any level of recursion, and then the situation becomes unified with Dmytry's concern.
If you were to make even considering such solutions have severely negative utility and somehow prevent pragmatic modifications you are effectively reducing its available solution space. The FAI has a penalty that an uFAI doesn't. Potential loss to an uFAI may always have a higher negative utility than becoming more pragmatic. The FAI m... (read more)
Doesn't this merely meta sidestep the issue? Now what the AI needs to do is modify itself to use pragmatic goals when modifying itself, to any level of recursion, and then the situation becomes unified with Dmytry's concern.
If you were to make even considering such solutions have severely negative utility and somehow prevent pragmatic modifications you are effectively reducing its available solution space. The FAI has a penalty that an uFAI doesn't. Potential loss to an uFAI may always have a higher negative utility than becoming more pragmatic. The FAI m... (read more)