A predicted difficulty of advanced safety is asserted to be "patch-resistant" or a "resistant problem" if it is asserted that proposed simple solutions ("patches") will fail to solve the difficulty and just regenerate the difficulty somewhere else.

Definition.

The notion of a problem being "patch-resistant" is relative to the current state of the art. If a speaker asserts a problem is "patch-resistant" and is using that term as defined here, it means the speaker thinks both of the following:

1. The speaker has heard 'simple' solutions proposed to the difficulty, or sees naive solutions that might be imagined, and the speaker thinks that in all of these cases the difficulty regenerates despite the naive solution.
2. The speaker thinks that relative to the current state of the art, we don't have a simple way to resolve the problem, or that proposed methods to resolve the problem don't yet have acceptable proposed implementations. (In other words, besides some naive solutions failing, there's also no known simple correct solution.)

Thus the NearestUnblockedNeighbor problem may be called "patch-resistant" by someone who thinks that (1) it is possible to design AdvancedAgents with a FullCoverage goal system, (2) a FullCoverage goal system would negate most NearestUnblockedNeighbor problems, but (3) we don't yet know how to build a FullCoverage goal system.

To call a problem "patch-resistant" is not to assert that it is unsolvable, but it does mean the speaker is cautioning against naive or simple solutions and that the speaker thinks some larger research project is required to deal with it.

Arguments.

Todo

Patch-resistant problems are common in AI and AGI generally.

See e.g. the Todoof nonmonotonic logic intended to represent 'explaining away' evidence, before the invention of Bayesian networks, as recounted in e.g. cite[Probabilistic Reasoning in Intelligent Systems].

See also http://lesswrong.com/lw/l9/artificial_addition/.

Patch-resistant problems arise in value alignment because of the complexity of value.

Todo this is why we can't have the simple amazing formal criterion that solves everything

Patch-resistant problems arise with strong optimizers because of how high-scoring 'bad' solutions often have high-scoring neighbors

Todo

• Patch-resistant problems arise with consequentialist agents because when you block one path, they seek another path
• Patch-resistant problems arise with reflective agents because if you change the code, you may not have changed the instrumental pressure to change the code back