I did a bachelors in math at an elite US university, and was exceptionally good at math even compared to my peers, so my observations may well be totally useless. With that out of the way:

I think it depends very much on what you expect to get out of math. On one end of the spectrum, if you want to be Gauss, you're almost certainly out of luck. If that were in the cards, you would have at some point realized that you're the smartest person almost everyone you know has ever met. On the other end, there's nothing in a standard undergraduate math curriculum that an average person couldn't understand, with enough time and effort.

Regardless of whether you fall on that spectrum, I think it's a terrible idea to actually use your IQ to decide whether you should study math- no matter how good a predictor it is, your actual math performance is an even better one. Every single person I know who struggles with math began struggling immediately on contact with real proofs. If you can handle that, even with some degree of difficulty, you're fine.