Patch resistance

A predicted difficulty of advanced safety is asserted to be "patch-resistant" or a "resistant problem" if it is asserted that proposed simple solutions ("patches") will fail to solve the difficulty and just regenerate the difficulty somewhere else.

Definition

The notion of a problem being "patch-resistant" is relative to the current state of the art. If a speaker asserts a problem is "patch-resistant" and is using that term as defined here, it means the speaker thinks both of the following:

1. The speaker has heard 'simple' solutions proposed to the difficulty, or sees naive solutions that might be imagined, and the speaker thinks that in all of these cases the difficulty regenerates despite the naive solution.
2. The speaker thinks that relative to the current state of the art, we don't have a simple way to resolve the problem, or that proposed methods to resolve the problem don't yet have acceptable proposed implementations. (In other words, besides some naive solutions failing, there's also no known simple correct solution.)

Thus the Nearest unblocked strategy problem may be called "patch-resistant" by someone who thinks that (1) it is possible to design an advanced agent with a full coverage goal system, (2) a full coverage goal system would negate most Nearest unblocked strategy problems, but (3) we don't yet know how to build a full coverage goal system.

To call a problem "patch-resistant" is not to assert that it is unsolvable, but it does mean the speaker is cautioning against naive or simple solutions and that the speaker thinks some larger research project is required to deal with it.

Arguments

Patch-resistant problems are common in AI and AGI generally

See e.g. the history of nonmonotonic logic intended to represent 'explaining away' evidence, before the invention of Bayesian networks, as recounted in e.g. Probabilistic Reasoning in Intelligent Systems

See also http://lesswrong.com/lw/l9/artificial_addition/.

Patch-resistant problems arise in value alignment because of the complexity of value

Patch-resistant problems arise with strong optimizers because of how high-scoring 'bad' solutions often have high-scoring neighbors

• Patch-resistant problems arise with consequentialist agents because when you block one path, they seek another path
• Patch-resistant problems arise with reflective agents because if you change the code, you may not have changed the instrumental pressure to change the code back