What about food? Which has arguably a much bigger effect on your hormones, mood, and lived experience? Will you commit to a bland and static diet to remain true to yourself?
What about illness or injury, and what about trauma? Will you commit to an ultra-low-risk lifestyle to avoid your "true self" being altered by adverse experiences?
What about sleep loss and tiredness? Will you commit to a strict sleep schedule? Are you capable of that without the use of substances?
What about the way successes and failures and the way they make you feel? Will you commit to a state of not-doing to preserve yourself from the dopamine highs and lows from achieving or not achieving your goals?
Anywhere you draw the line seems incredibly arbitrary to me. Life will affect your hormones and neurotransmitters, which will in turn affect your personality, moods, affect, and experience. Unless you plan to become a monk and remove yourself from stimulus, you can't avoid this. It makes sense to me to avoid doing highly mind altering substances and activities when you need to make a big decision or have a clear head, but I don't see any sense in abstaining completely or especially in drawing a line at sex but not food.