while the actual representatives of EA/rationalist community probably don't even notice that this happens

I think it matters a lot whether this is true, and there is widely known evidence that it isn't true. For example Brent Dill and (if you are willing to believe victims) Robert Lecnik. 

Your post is well said and I am also very worried about EA/rat spaces as a fruitful space for predatory actors.