I think a big hurdle in the way at the moment is the convergent abstractions hypothesis, if true.  As a concrete example, we know that RLVR increases metagaming, including modelling the grader, and I expect RLVR after SOO fine-tuning will bring back the SO distinction. 

I have been thinking about whether representation-level fine-tuning is enough, or if we need to completely rethink the incentive structures in LLM training for self-other distinctions to disappear. An out-there solution, for example, is to train LLMs as part of a multi-agent system (close to a hive mind), of which some agents are humans and other sentient beings we care about, optimising for the values (or, if we want corrigibility, preferences) of agents within. It's unclear to me whether such training setups avoid some x-risk for the non-LLM sentient agents within, especially as the LLM agents become vastly more superhuman.