Edit: TL;DR: Mathematics is largely Ra worship, perhaps worse than even the more abstract social sciences. This means that That Magic Click never happens for most people. It's a prime example of "most people do not expect to understand things", to the point where even math teachers don't expect to understand math, and they pass that on to their students in a vicious cycle.

Surely as soon as you see the formula ... you know that you are dealing with some notion of addition that has been extended from the usual rules of addition.

Only if you know that it's possible to have multiple rules of addition. That's an unknown unknown for almost everyone on the planet. Most people aren't even familiar with the concept of unknown unknowns, and so are hopelessly far away from this in idea space. For them, they are more likely to just reject logic and math entirely as obviously wrong.

That requires being aware of the fact that addition can be constructed in multiple ways, which is very much NOT something you learn in school. They basically just present you with a series of weird looking "facts", and give a handwaving explanation. I suspect the vast majority of people, maybe even a narrow majority of LessWrongers, wouldn't even know that disagreeing with mathematics is something you're allowed to do. (“It’s math, it’s totally unambiguous, you can’t just disagree about the results.”) I suspect that's why this post has as many upvotes as it does, even if most of us are dimply aware of such things.

Let me try and explain where I'm coming from with this. I don't know about the rest of you, but I always went through the exact same procedure after learning each new layer of mathematics. It goes something like this:

Phase 1: Wait, 1234x5678 can be solved by multiplying 4x8, then 4x70, then 4x600, ect., then adding it all up??!! What are the chances of that algorithm in particular working? Of all the possible procedures, why not literally anything else?

Phase 2: Ok, I've done some simple examples, and it seems to produce the correct result. I guess I'll just have to grudgingly accept this as a brute fact about reality. It's an irreducible law that some ancient mathematician stumbles upon by accident, and then maybe did some complex an impenetrable sorcery to verify. Maybe someday I'll get a PhD in mathematics, and maybe then I'll understand what's going on here. Or maybe noone really understands it, and they just use a brute force solution. They just try every possible algorythm, in order of increasing kolmogrov complexity until one works. Pythagoras tried A+B+C=0, A+B=C, etc until finding that A^2 + B^2 = C^2. Progress in mathematics is just an automated, mechanical process, like supercomputers doing things entirely at random, and then spitting out things that work. No one really understands the process, but just blindly applying it seems to produce more useful math theorems, so they keep blindly turning the crank.

So, upon being told that A^2 + B^2 = C^2, or that 1+2+3+4+5+… = -1/12, my initial reaction is the usual disbelief, but with the expectation that after an hour or two of toying with numbers and banging my head against the wall trying to make sense of it, I'll invariably just give up and accept it as just one more impenetrable brute fact. After all, I've tried to punch holes in things like this ten thousand times before and never had any success. So, the odds of making any sense of it this time can't be more than 0.01% at most, especially with something so far above my head.

How can someone even do math without understanding what math is? Well, I can only offer my own anecdata:

I was always good at math through highschool, but I suspect I spent twice as much time as everyone else doing the homework. (When I did it. I didn't bother if I could get A's despite getting 0's on my homework.) Most of this time was spent trying to decipher how what we were doing could possibly work, or solving the problems in alternate ways that made more sense to me.

When I hit Calculus in college, I promptly failed out because I didn't have enough time to do the homework or complete the tests my way. (I rarely just memorized formulas, but instead beet my head against the wall toying with them until I more or less knew the algorithm to follow, even if I didn't understand it. I didn't know about spaced repetition yet, so I was unable to memorize enough of the formulas to pass the tests, and didn't have time to derive them.)

I concluded that I was just bad at math, especially since I could never follow anything being written on the board, because I would get stuck trying to make sense of the first couple lines of any proof. I considered my mathematical curiosity a useless compulsion, and assumed my brain just didn’t work in a way that let me understand math. In retrospect, I don't think anyone else in any of the classes actually understood either, but were just blindly following the algorithms they had memorized.

Personally, I have acquired 3 clues that math isn't just a series of random brute facts:

1. Philosophy of Mathematics has a divide between Mathematical Platonism and Empiricism. I was really confused to hear a calculus professor make an offhand empiricist remark, because I wasn't aware that there was an alternative to Platonism. I had always just assumed that math was a series of platonic ideal forms, suspended in the void, and then physics was just built up from these brute facts. The idea of math as a social construct designed to fit and understand reality was bizarre. It wasn't until I read Eliezer's The Simple Truth and How to Convince Me That 2+2=3 that it really clicked.

2. I stumbles upon A Mathematician's Lament, and gained a bunch of specific insight into how new mathematical ideas are created. It's difficult to sum up in just a few words, but Lockhart argues that how we teach mathematics would be like teaching music by having kids memorize and follow a vastly complex set of musical rules and notations, and never let them touch an instrument or hear a note until graduate school. After all, without the proper training, they might do it wrong. He argues that mathematics should be a fundamentally creative process. It is just a bunch of rules made up by curious people wondering what would happen to things if they applied those rules. Previously, whenever I saw a new proof, I'd spend hours trying to figure out why they had chosen those particular axioms, and how they knew to apply them like that. I could never understand, and figured it was way beyond my grade. Lockhart provides a simple explanation, which has since saved me many hours of handwringing: They were just playing around, and noticed something weird or cool or interesting or potentially useful. They then played around with things, experimenting with different options to see what would happen, and then eventually worked their way toward a proof. Their original thought process was nothing like the mysterious series of steps we memorize from the textbook to pass the test. It was exactly the sorts of things I was doing when I was toying with numbers and formulas, trying to make sense of them.

3. I recently taught myself some lambda calculus. ("Calculus" here doesn't mean integration and differentiation, but only the simplest forms of operations. In fact, the basics are so simple that someone made a children's game called Alligator Eggs out of the rules of lambda calc.) It's basically just a simple set of rules, that you can string together and use to build up some interesting properties, including AND, OR, IF, IFF operators, integers, and addition/subtraction.

Let me tie it all back together. Apparently there are multiple ways of building up to operators like this, and lambda calc is just 1 of several possibilities. (And, I would have been mystified as to why the rules of lambda calc were chose if it weren't for reading The Mathematician's Lament first.) Under the mathematical empiricist view, by extension, it's not just how we build up to such operators that's arbitrary. It's ALL OF MATHEMATICS that's arbitrary. We just focus on useful operators instead of useless ones that don't fit reality. Or not, if we find other things interesting. No one expected non-Euclidian geometry to be useful, but as it turns out spacetime can warp, so it drifted into the domain of applied mathematics. But it started as someone toying around just for lolz.

Well, Numberphile says they appear all over physics. That's not actually true. They appear in like two places in physics, both deep inside QFT, mentioned here.

QFT uses a concept called renormalization to drop infinities all over the place, but it's quite sketchy and will probably not appear in whatever final form physics takes when humanity figures it all out. It's advanced stuff and not, imo, worth trying to understand as a layperson (unless you already know quantum mechanics in which case knock yourself out).