Doesn't this merely meta sidestep the issue? Now what the AI needs to do is modify itself to use pragmatic goals when modifying itself, to any level of recursion, and then the situation becomes unified with Dmytry's concern.
If you were to make even considering such solutions have severely negative utility and somehow prevent pragmatic modifications you are effectively reducing its available solution space. The FAI has a penalty that an uFAI doesn't. Potential loss to an uFAI may always have a higher negative utility than becoming more pragmatic. The FAI may evolve not to eliminate its ideals but it may evolve to become more pragmatic, simplifying situations for easier calculation else be severely handicapped against an uFAI.
How can you prove that in time the FAI does not reduce to an uFAI or be quickly rendered to a less steep logistic growth wrt uFAI? That its destiny is not to be come an antihero, a Batman whose benefits are not much better than its consequences?
Doesn't this merely meta sidestep the issue? Now what the AI needs to do is modify itself to use pragmatic goals when modifying itself, to any level of recursion, and then the situation becomes unified with Dmytry's concern.
If you were to make even considering such solutions have severely negative utility and somehow prevent pragmatic modifications you are effectively reducing its available solution space. The FAI has a penalty that an uFAI doesn't. Potential loss to an uFAI may always have a higher negative utility than becoming more pragmatic. The FAI may evolve not to eliminate its ideals but it may evolve to become more pragmatic, simplifying situations for easier calculation else be severely handicapped against an uFAI.
How can you prove that in time the FAI does not reduce to an uFAI or be quickly rendered to a less steep logistic growth wrt uFAI? That its destiny is not to be come an antihero, a Batman whose benefits are not much better than its consequences?