The anti-naturality problems are an issue, especially if you want to build the thing via standard RL-esque training, but they're not the first things which will kill you.

The story in the post you link is a pretty standard training story, and runs into the same immediate problems which standard training stories usually run into:

• The humans will feed the system incorrect data.
• Insofar as the system is capable, it will learn to predict the humans' own errors, as opposed to the thing the humans intended (and this will get worse as capabilities increase).
• Insofar as the incorrect data stems from the humans being bad at understanding what they should ask for, the resulting problems will be systematically hard to notice; things will look correct to the humans.

These problems basically don't apply in domains where the intended behavior is easily checkable with basically-zero error, like e.g. mathematical proofs or some kinds of programming problems. These problems are most severe in domains where no humans understand very well what behavior they want, which is exactly the case for corrigibility.

So if one follows the training story in the linked post, the predictable result will be a system which behaves in ways which pattern-match to "corrigible" to human engineers/labellers/overseers. But those engineers/labellers/overseers don't actually understand what corrigibility even is (because nobody currently understands that), so their pattern-matching will be systematically wrong, both in the training data and in the oversight. Crank up the capabilities dial, and that results in a system which is incorrigible in exactly the ways which these human engineers/labellers/overseers won't detect.

That's the sort of standard problem which trying to train a corrigible system adds, on top of all the challenges of just building a corrigible system at all.