This is anecdotal, but it's also the primary reason I'm not worried.

A friend of a friend heard about the phytoestrogen thing, and wanted to transition from male to female, so attempted DIY hormone therapy by eating a large quantity of soy products, as in, several ounces of soy at every meal, drinking the liquid they pack tofu in, drinking lots of soy milk, etc.

There was no noticeable effect after months of trying.

Obvious caveats on this story for n=1, but additionally given the high prevalence of soy in the world, my read is that either even a little soy is a problem and we don't see clear effects of eating more soy because it's already had most of its impact on us or it's just not that big a concern. Either way, unclear that anything short of extreme action to avoid soy would be likely to show an effect if there is one, and then we'd have to worry about confounders from whatever those extreme actions are.

According to this plant-based-leaning but also somewhat vegan-critical blog led by a sports nutritionist, eating 4 doses of soy per day (120 g of soybeans) is safe for male hormonal balance. It's in Polish but Google translate should handle. He cities studies. https://www.damianparol.com/soja-i-testosteron/

My understanding is phytoestrogens are plant hormones. If you’d like to intake more animal oestrogen I believe cows milk contains quite a bit.

Anecdotally, I’ve been drinking soy milk every day for ~2 years now. I sometimes go for a week or two eating Tofu as my primary food. I really don’t think it’s a problem.

I however don’t want to comment on soy oil. I don’t have enough info about that.