(Take with a grain of salt, I'm far from IMO level and never seriously trained for math contests.)
After solving any given problem, reflect on general methods that would allow solving a bigger class of problems including the one you've cracked. For any miracle of intuitively seeing a solution method (or noticing some useful property), look for ways of more systematically inferring a workable method that don't rely on miracles. Don't consider a problem solved just because you solved it (i.e. used your intuition), you should also figure out how it could be solved (i.e. know in more detail how your intuition figured it out, or know a method other than the unknown one used by your intuition).
I expect this can get one past some limitations of raw ability that wouldn't otherwise be lifted using just problem-solving practice, but I'm not sure how far.
This is a reasonable portion of what I did for math olympiads; the other parts were doing lots and lots of problems and acquiring a solid technical background.
One thing I worked on in particular is formulating good solution strategies, where I could see the general steps of a solution (or the steps of a good approach) without having to actually fill in all the details of the approach; this involves having good heuristics for figuring out what is true/false and what can be proved without too much effort (and then deferring the actual proof until later when all the pieces have come into place).
FTR I was a USAMO winner (top 12) in high school but didn't make it to the IMO team. I did make the IOI team though. I'm currently coaching at the US IOI training camp for the next week, so I don't have much spare time right now, but maybe when I have some I'll share in a bit more detail the things I did (just a warning, this was before I started optimizing my behavior so while it apparently worked it was probably not an optimal trajectory).
(I expect top programming contests are harder/less feasible to train for at levels outside raw ability. It all happens too fast.)
I expect this can get one past some limitations of raw ability that wouldn't otherwise be lifted using just problem-solving practice, but I'm not sure how far.
A lot farther than most people realize. Few people actually try going meta, because social structures don't encourage it. (At least not in a near sense.)
Not going meta for developing reliability of problem-solving took a lot of points from me. I just relied on the magical intuition, which was good enough to solve some hard problems (to figure out solution method, without knowing how it was being figured out), but not good enough to reliably solve those problems without errors.
As a result, when I was applying to college, I was afraid of the regular admission exams which I couldn't reliably ace (because of technical errors I wouldn't notice, even though solution methods were obvious), and instead used the perfect score given to winners of Moscow math and physics olympiads, which required solving some hard problems but not solving all problems without errors. Which is a pretty stupid predicament. It just never occurred to me that production of perfect scores can be seen as an engineering problem, and I don't recall any high school teachers mentioning that (even smart college professor teachers administering cram school sessions).
What do you mean by "going meta"?
I feel like the advice in your earlier comment is good for obtaining insight, but I can't see how it would be useful on a test. I haven't taken many tests where I have had enough time to solve each problem in several ways!
I'm eager to learn more if I haven't understood correctly, though.
For harder tests, the benefit is in not ignoring low-hanging fruit, and training to look for any opportunity to get better reliability, performing cheap checks and selecting more reliable of any alternative sub-steps. On the other hand, ordinary exams are often such that a well-prepared applicant can solve all problems in half the time or less, and then the failure would be not taking advantage of the remaining time to turn "probably about 90% of solutions are correct" into "95% chance the score is perfect".
Learn to convert time into reliability of a solution. Don't just solve a problem (in the contest setting), but check correctness of the solution from as many angles as remaining time allows.
Generalize the problem, solve in general, check that the general solution gives the same answer on edge cases as straightforward solutions in those cases. Solve using a different method. Infer additional facts about the problem that it doesn't ask you to infer, and infer from those facts other facts you encountered. Reduce the problem or parts of the problem to different formulations, solve in those different formulations, translate back, check that it fits. Invent redundant overlapping subproblems just to compare intermediate results. (Whichever of these is most natural.)
(This is the greatest piece of low-hanging fruit that I never collected. In particular, knowing this trick of converting time into reliability allows to get perfect scores on simpler tests and not lose points for harder problems that you know how to solve in principle when there's enough time.)
For the USAMO/IMO, the ability to solve a single problem is already a large step. Solving 4 out of the 6 problems means you almost surely make the IMO team. I think the issue is more how he can get good enough to solve such difficult problems (they are also proofbased, so an actual solution should always get full points barring a bad writeup).
One core skill is ability to solve problems eventually, to not stop before you solved the problem. Bringing this to contest setting requires speeding up to a level that is probably unattainable because of limitations of raw ability, and might benefit from lots of training. But another failure mode is stopping right after you solve the problem, so that you won't notice a stupid error, even though you've got the solution. This is a piece of low-hanging fruit that doesn't necessarily require any significant effort (in comparison), but still needs to be collected.
The math olympiad is not like most tests - each problem has one or more key insights you must have in order to solve it. Get the insight; solve the problem. Don't have the insight, don't solve the problem.
This piece of advice was for preventing "get the insight, solve the problem incorrectly by making a stupid technical error".
Sorry not to have answered you earlier, I have been absent for the first days in the final round of selection for my country's IMO team, which after years of practice I am finally on. Feel free to PM me if you have more questions, though I can't promise that I'll know all the answers.
As far as practice vs learning new maths goes, the IMO has something of an unwritten syllabus of theorems you need to know. If you don't know everything on it, then by far your best route is to learn the things on it. Once you've got that, learning more theorems is unlikely to help much, and you just need practice, practice and more practice. I have tried to solve at least one problem every day for the past year.
The one exception is that if you start to find that you simply lack the raw talent required in one ore more of the areas, then try learning unconventional techniques for doing it, you may be better at those.
If you're not sure which stage you're at yet, try some problems on Art Of Problem Solving, then look at the solutions of the ones you couldn't do within a day or so. If you understand all the solutions then you probably have a sufficient knowledge base.
A trap to avoid falling into is only practising the areas you are good at, this is a very seductive mistake since inevitably those areas will feel more fun to do. Try the areas that seem hard and boring until they become fun, but if you start to find yourself disliking maths in general switch back to something you enjoy.
If possible try to find a mentor of some kind, someone you can meet in real life is best. This is especially essential if you're still at the stage where you don't know the whole unwritten syllabus yet.
This type of approach may not be the best if you actually wish to become a good mathematician, but the IMO is sufficiently competitive that if you are unlikely to get on with anything less (I only made it by the tiniest margin as it is).
Concerning nootropics, for the most part what little evidence there is for them working is measured on people that have aged or damaged brains.
Pop quiz. This was one of my favorite questions that I recall back in my middle school math team days (it's probably easy for a mathy adult though): "What is the length of a line that spirals once around the surface of a cylinder as it descends from top to bottom?"
I remember that problem from one of Gardner's books. It's one good example of how generalizing can help you avoid pitfalls - try more than one spiral, or shapes other than cylinders.
Do math contest problems from the different areas of math covered by the IMO and start at your current level of ability and work from there.
Find problems from books, previous contests and forums like Art of Problem Solving.
In terms of book recommendations, if you're interested, I highly recommend "Problem Solving Strategies" by Arthur Engel. In addition to being useful in general, it divides olympiad problem strategies in what I believe are natural categories; you could work on teaching yourself to recognize, by looking at a problem, which broad types of strategies are likely to work for it, and get yourself closer to a solution that way.
It also helps to look at all past problems in a specific area of math, and see the specific results and theorems that they use. Then you can approach all problems in the area with those tools in mind. This has helped me a great deal when preparing for the Putnam (I used it for linear algebra). It will be more helpful with some areas than others (for geometry more than combinatorics, for instance).
Lately I've resolved to try harder at teaching myself math so I have a better shot at the international olympiad (IMO). These basically involve getting, say, three really hard math problems and trying your best to solve them within 5 hours.
My current state:
What does the intrumental-rationality skill of LWers have to say about this? What recommendations do you guys have for improving problem-solving ability, in general and specifically for olympiad-type environments? Specifically,