Is that a very good argument though? To believe that there is a reason to do something simply because lots of people do so, sounds like a bias to me...

I feel that the sentence

Suppose you say that you're 99.99% confident that 2 + 2 = 4. Then you have just asserted that you could make 10,000 independent statements, in which you repose equal confidence, and be wrong, on average, around once.

is a little questionable to begin with. What exactly is an "independent" statement in this context? The only way to produce a statement about whether 2 + 2 = 4 holds, is to write a proof that it holds (or doesn't hold). But in a meaningful mathematical system you can't have two independent proofs for the same statement. Two proofs for the same thing are either both right or both wrong, or they aren't proofs in the first place.