A lot depends on your model of progress, and whether you'll be able to predict/recognize what's important to understand, and how deeply one must understand it for the project at hand.

Perhaps you shouldn't frame it as "study early" vs "study late", but "study X" vs "study Y". If you don't go deep on math foundations behind ML and decision theory, what are you going deep on instead? It seems very unlikely for you to have significant research impact without being near-expert in at least some relevant topic.

I don't want to imply that this is the only route to impact, just the only route to impactful research.
You can have significant non-research impact by being good at almost anything - accounting, management, prototype construction, data handling, etc.