Take a look at this multiple choice question from the Open University BSc on psychology. Can you tell what is wrong with it. This is sampled from a set of 15 questions, some of the others aren't this bad, some are guessing the teachers password. One of the questions requires that you know that statistically significant means p<0.05 and can look at a table of p values and say which numbers are less than 0.05.
An educational psychologist believes that an individual's reading speed (in words per minute) and general comprehension abilities (scored out of 10) may be related to how well they do on a timed exam. To study the relative importance of these variables and the association between them, she runs a multiple regression and produces the following predictive model for exam score:
Exam Score =7.78 + (0.05*Reading Speed)+(5.77*Comprehension)
What exam score would you expect students to achieve if they have a reading speed of 120 w.p.m and a comprehension score of 5?
What level of understanding is necessary to fully understand the question?
From a mathematicians perspective, Multiple linear regression is an algorithm based on variational calculus (Optimizing over a space of functions), to select the function that maximizes the probability density that a model consisting of a linear function from the vector space to plus a Gaussian random variable (with arbitrary standard deviation, that term happens to cancel out) assigns to the observed data. This involves function spaces, matrix inversions, multivariate integrals ect. To understand the mathematical structure that leads up to multivariate regression, you need the idea of jacobians and vector calculus just to integrate your to find the normalizing constant on your Gaussian random variables. You need continuus probability spaces, but probability isn't defined over arbitrary sets of real numbers, so you need sigma algebras, which brings you to set theory. You need to prove that the integrals exist, that the optimum slope exists and is unique, ect.
In short you need to understand a lot of maths to deeply understand what is going on.
How much understanding do you need to get the answer right. Suppose you have basic English comprehension, but all you know about the symbols "385+*)" is that they are some mysterious code that you can type into a calculator and it will reply in the same code.
You know that the answer has to be in this mysterious code, and the calculator doesn't have buttons for letters. You know its asking for an exam score, so you type in the code for "Exam score", but replacing the word "Reading speed" with the mysterious symbols "120", and the word "Comprehension" with the symbol "5".
Your calculator spits out an answer, it is one of the options available, you put it down. (You get an exact answer, no rounding needed)
Lets suppose your grasp of English was little better than your grasp of arithmetic.
You have no idea what an "educational psychologist" is.
You have a rough grasp of capital letters, you know that the first letter in words sometimes looks big or funny, enough to suspect that "Reading Speed" and "reading speed" are the same thing. None of the mysterious numbery things in the question match any of the answers, so you try typing the big numbery thing into a calculator. The calculator doesn't have any lettery buttons, so you replace the lettery things with the numbery things that are near them in the text. If you guess the wrong numbery thing the first time, you don't get an answer on the list, so if you know what a multiple choice test is, you will know to try another.
Maybe I am being a little scathing there, I am certainly assuming that the hypothetical person answering the question knows some exam technique, and are able go along with plausible guesses. But I would expect most people who knew basic English and arithmetic to be able to answer it correctly.
The basic problem is
An exam question only measures if you have enough understanding to answer it correctly.
The amount of understanding needed to answer a question can be lower than it appears.
The question references all sorts of advanced maths, giving the appearance of serious academic learning, but instead of asking you to show an understanding of that maths, it asks you something much simpler.
This is like guessing the teachers password, except that the password is a procedure, not a direct answer.