As you've noted, Bayes' Theorem is just a straight forward result of probability calculus. In that light, it is entirely uncontroversial.
What people really seem to get excited about is Bayesianism, which is something more than just the application of Bayes' Theorem.
To understand people's interest in Bayesianism, I think you then need to distinguish its use in two types of applications: how we use probabilities to deal with uncertainty when drawing inferences from data generated by scientific studies (i.e. statistical inference); and whether humans reason/learn, or should reason/learn, in a Bayesian manner.
The latter would be well outside my own expertise, but I once got a fair number of interesting responses to this question from people that would know better than I. Regarding its use in statistical inference, Bayesianism is similarly controversial, and the many controversies are the subject of hundreds and thousands of papers and books.
Randomized experiments (including clinical trials) are widely recognized as useful tools for trying to understand cause and effect. They are also criticized, both fairly and unfairly. Some of the more common misconceptions around randomization follow from the idea that randomization is useful because it "balances confounders", but this is not correct..not exactly anyway. The linked essay tries to explain why it's not correct, why it matters, and the implications this misconception has for the design of clinical trials.
Another key work here is Probability Theory: The Logic of Science by ET Jaynes. (you can download the entire book here). The early chapters are focused on deriving the probability calculus from logic.