I don't even know how to answer this because it's coming from a place that's so foreign to me. I have a quite weak visual imagination (not full-on aphantasia), and I've never heard a voice speaking words in my head when I read (although actually, now that I've listened to a lot of audiobooks, I can force this to happen briefly if I concentrate). But I've always enjoyed reading! To me, I guess I would say, words are just sort of fundamental? Like, the word itself, the shape of squiggles on the page, is the thing that has meaning, and I don't have to visualize anything beyond it to understand it. It's like the difference between reading a foreign language you're not that good at, where you still have to translate every thought into your native language to really understand it, versus reading in your native language, where there's no translation step.
It might be true, as quoted in Mo's comment below, that people with weak visual imaginations are less likely to enjoy extremely visual-description-heavy fiction like Lord of the Rings – I indeed found reading the LotR books mind-numbingly boring, borderline painful. But in almost all works there's a ton of content that can be enjoyed without imagining it into being: concepts, emotions, even most kinds of events. I also find myself very drawn to beautiful writing, wordplay, and just skillful use of language in general, and now that I think about it, this is probably why. Neat!
The same applies to nonfiction – building an understanding of the connections between concepts doesn't have to rest on any sort of visual framework. The concepts can just become connected, literally/physically, in the structure of your brain.
I do think that I would have struggled less with university-level physics and the related math if I had more of a visual imagination. It's quite hard to keep track of things when it all just feels like symbols you're manipulating; I imagine that being able to visualize things would have let me viscerally understand connection between the math and the physical systems being described. As it was I just sort of, knew explicitly that they were connected. But it was all very vague and confusing.
I hope this helps you understand the experience somewhat. Feel free to ask me followup questions.
Oh, and I also notice that despite my weak visual imagination, movie adaptations can still 'ruin' books for me, not exactly because they lock in a certain way that things look, but because they lock in the characters' personalities and the general vibe.