Suppose this is true, and we eventually find out what each neuron in the networks correspond to and can ablate the ones that are misaligned to prevent reward hacking / emergent misalignment. Doesn't this equally increase the capability of anyone to ablate the opposite pole i.e. the ones that cause 'aligned' behaviour? What's going to stop anyone from doing that when we reach ASI level intelligence? This also seems like it improves capabilities research, so is it possible that this leaves us worse off than when we started?
Suppose this is true, and we eventually find out what each neuron in the networks correspond to and can ablate the ones that are misaligned to prevent reward hacking / emergent misalignment. Doesn't this equally increase the capability of anyone to ablate the opposite pole i.e. the ones that cause 'aligned' behaviour? What's going to stop anyone from doing that when we reach ASI level intelligence? This also seems like it improves capabilities research, so is it possible that this leaves us worse off than when we started?