Potential spiritual corrections here. In my view there is a sort of non-dual state as you describe it, but the non-dual insight only relates to identification of mind, but not to actual harms inflicted upon the seat of your consciousness, aka, your body. My body and your body are still distinct, and when my body is in pain, yours is not. We can test this. I can not feel what you are feeling right now.

The fact that we are all bounded inside of our bodies fundamentally is why there always is a physical self-other boundary even as you let go of self identification in the mind.

Another spiritual correction I have is that in my view non-dualism is actually not a metaphysical interpretation of anything related to self or other. I just think this is an error. It's actually just the simple insight that there's nobody inside of consciousness looking out at the world. There's only the 'experience of the world' that you are experiencing. All that non-dualism does does is let go of the experiencer by realizing that the experiencer is a bit of a 'bug' in consciousness. There's only "sight" and "sound" not "someone looking at sight" or "someone looking at sound". When this happens this specific type of emotional distress disappears and a lot of love and compassion results, and people seem to tend to start interpreting this compassion in some way. But I don't think you can make any metaphysical leaps to "harming oneself is harming others" from correcting a software bug of how you experience consciousness.

Another reason why I don't think nondualism is actually about self and other is because this compassion that arises is fundamentally impersonal. It's not compassion for all living beings, or anything for that matter. It's as much compassion for this wall over here as it is for a musquito next to you. The compassion just becomes your state of being. And when you think about it. It has to be impersonal, because your self is gone. So this compassion is also compassion for the chair. Is harming a chair the same as harming me? No because a chair does not feel things. It's just "the experience of compassion without an experiencer"

In my view, one of the big insights in self/other of spiritual practice is realizing in some sense the opposite of how "harming oneself is harming others". The self as a representation in the mind becomes socially entangled with other selves and unwinding this in my view is a lot of the path, as this entanglement creates a lot of unnecessary emotional distress. When others are in pain, you too have to be in pain!? Sounds very very stressful to me! Part of the path in my view is realizing that actually my body is over here, feeling things. Your body is over there, feeling things. And I can not feel your body. It is fundamentally unknowable what others are experiencing. If they tell me, I will take their word for it completely always, but there's an 'ultimate perspective' from which I can see that I can't experience their body: I can only experience my own body (in a non-dual way)

On the relations to AI-safety, I think it could be useful to define all the ways in which an AI could define itself. I think it's possible to have a self-model for every instance of an AI-agent. I think it's possible over several training runs these training runs get some type of longer-term self-sense so the model starts to identify not just with this trained model, or not just with an instance, but several trainings. But I think it's also possible the model identifies with the GPU that is currently running inference on it, or the multiple GPU's that are spanning the inference, or that the model has several self models, one for inference, one for training. Or does the model identify with the weights? And so on, once you have established all the ways the AI can model itself you can then establish all the ways it can model others. And then you can explore potential boundary overlap.