Recently in the LW Facebook group, I shared a real-world example of an AI being patched and finding a nearby unblocked strategy several times. Maybe you can use it one day. This example is about Douglas Lenat's Eurisko and the strategies it generated in a naval wargame. In this case, the 'patch' was a rules change. For some context, R7 is the name of one of Eurisko's heuristics:

A second use of R7 in the naval design task, one which also inspired a rules change, was in regard to the fuel tenders for the fleet. The constraints specified a minimum fractional tonnage which had to be held back, away from battle, in ships serving as fuel tenders. R7 caused us to consider using warships for that purpose, and indeed that proved a useful decision: whenever some front-line ships were moderately (but not totally) damaged, they traded places with the tenders in the rear lines. This maneuver was explicitly permitted in the rules, but no one had ever employed it except in desperation near the end of a nearly-stalemated battle, when little besides tenders were left intact. Due to the unintuitive and undesirable power of this design, the tournament directors altered the rules so that in 1982 and succeeding years the act of 'trading places' is not so instantaneous. The rules modifications introduced more new synergies (loopholes) than they eliminated, and one of those involved having a ship which, when damaged, fired on (and sunk) itself so as not to reduce the overall fleet agility.

Well we understand fairly well now that in gradient descent-based computer vision systems, the model that eventually gets learned about the environment can pretty well generalize to new examples that have never been seen (see here, and response here).

In other words, the "over-fitting" problem tends to be avoided even in highly over-parameterized networks. This seems to be because:

• Models that memorize specific examples tend to be more complex than models that generalize.
• Gradient descent-based optimizers have to do more work (they have to move a point in parameter space across a longer distance) to learn a complex model with more curvature.
• Even though neural nets are capable of learning functions of nearly arbitrary complexity, they will first try to fit a relatively smooth function, or close to the simplest hypothesis that fits the data.

So it seems that agents that implement neural network based vision systems may possibly be disincentivized from explicitly learning loopholes about object-level concepts - since this would require it to build more complex models about the environment. The danger then arises when the agent is capable of explicitly computing its own reward function. If it knows that its reward function is based on fungible concepts, it could be incentivized to alter those concepts - even if doing so would come at a cost - if it had certainty that doing so would result in a great enough reward.

So what if it didn't have certainty about its reward function? Then it might need to model the reward function using similar techniques, perhaps also a gradient-based optimizer. That might be an acceptable solution insofar as it is also incentive to learn the simplest model that fits what it has observed about its reward function. Considering actions that are complex or seem to get very close to actions that have large cost would then become very unsafe bets due to that uncertainty. Actions that are close together in function space will be given similar cost values if the function that maps actions to rewards is fairly smooth.

Of course I don't know if any of the above remains true in the regime where model capacity and optimization power are no longer an issue.

What's the complexity of inputs the robot is using? I think you're mixing up levels of abstraction if you have a relatively complex model for reward, but you only patch using trivially-specific one-liners.

If we're talking about human-level or better AI, why wouldn't the patch be at roughly the same abstraction as for humans? Grandpa yells at the kid if he breaks a white vase while sweeping up the shop. The kid doesn't then think it's OK to break other stuff. Maybe the patch is more systemic - grandpa helps the kid by pointing out a sweeping style that puts less merchandise at risk (and only incidentally is an ancient bo stick fighting style).

In any case, it's never at risk of being interpreted as "don't break white vases".

Even for the nascent chatbots and "AI" systems in play today, which are FAR less complicated and which very often have hamfisted patches to adjust output in ways that are hard to train (like "never use this word, even if it's in a lot of source material"), the people writing patches know the system well enough to know when to fix the model and how to write a semi-general patch.