I visualize math a lot. Ironically, I don't like geometry, because my visualization is not precise, I don't visualize lines and angles. My visualizations are more like: objects in 3D space; shapes in 2D space; or numbers written on a board.
For example, if you asked me to calculate from memory "464+245", I would keep repeating the numbers mentally (to keep them in the auditory loop), but at the same time I would imagine them written on the board below each other (ones below ones, tens below tens, etc.), and try to add them kinda like I would on paper.
When I think about set theory, I sometimes imagine the sets visually as graphs (small balls connected by strings), with the node representing the set at the top, and the nodes representing its elements below it, connected by lines; etc. Then e.g. the Axiom of Foundation becomes "there is no infinite downwards chain". Then I can use different colors for different types of sets, etc. Similarly, nonstandard integers are horizontal chains of small balls, etc.
I have a picture in my mind, then I read the axioms and translate them to statements about the picture. This sometimes allows me to make intuitive guesses, such as "it is impossible for a picture to be both X and Y" or "a picture that is both X and Y would have to look kinda like this", and then I try to translate the intuition back to the language of mathematical statements. Sometimes it turns out I was wrong, then I try to fix the picture.
It seems to me that the auditory and visual processing have different advantages and disadvantages. If I have to remember the number, the sound of "six - nine - eight" is more reliable than the picture of "698", because the sounds of individual digits are different, while the pictures are similar. On the other hand, the auditory loop is linear, and beyond certain size you have to use pen and paper; while visualization allows you to see 2D or 3D structures and mentally moving and rotating them, and make some statements that are simple in the visual form, but their description in words is clumsy. (It is easier to move your hands and say "rotate like this" than to say "rotate around the vertical axis by 90 degrees clockwise".)