That's an interesting question-- if you try testing it (I'm not sure how it would apply to phone numbers), please post.

I see a direct causal relationship.

Recall Piaget's sensorimotor model of intellectual development. This impressed me greatly in highschool because it stuck me as absurdly obviously true, but yet I had never heard it before. For example, you can't remember the word for scissors (this happens to me all the time) and so you pretend you are cutting paper with your index and middle fingers. This physical motion is literally a 'handle' for the concept 'scissors'. I believe that concepts are very frequently (if not always) tied to physical motions and physical space, and I believe (I believe Piaget) these concepts cannot be grasped until the physical development is there. The greatest piece of personal evidence for this is that I did not understand the concept of a variable until midway through my 17th year. Until then, abstract algebra was just a frustrating symbol manipulation. Once I understood what a variable meant, there was a new dimension there that wasn't there before. And it felt very much that this understanding just physically grew one day. I had tried very hard a few months before to understand, and wasn't able to make progress. In contrast, understanding the principle of induction was the result of continuous mental exercise over several years -- that understanding felt like the development of a muscle rather than the sudden emergence of one.

Back to positional gestures in conversation, I don't believe it's a coincidence and I often listen carefully to the words mathematicians use to gain insight into how something is "physically" understood. A function "lives" "in" a space -- for example. These aren't just sloppy anthromorophisms (though I do consider them sloppy), but the proper way a human being needs to handle them as objects in order to understand them.

Finally, when I think about it, I'm not at all surprised that we have two working memories, one on either side. This is because the brain has modeled remembering objects as "holding this in one hand" and "holding this in that hand" and maybe room for one more thing in each hand as well. (I realize this is speculation but I don't personally need an experiment to bear it out, since it just feels right enough. However, I'm ever open to information to the contrary.) I wonder if we could train our memory to work better by imagining objects dangling from our fingers. This would take some physical practice, in order to make the sense of each finger quite distinct. I wonder now about the working memory of pianists. It's certainly true that my fingers remember one of my login IDs, where my brain doesn't.