I used to teach physics to pre-med students (a nearly-identical situation to [2] in the original post). I tried to write my exams so that simply memorizing a large set of very specific algorithms for solving a problems wouldn't work, but nobody would have to be very clever in order to get a good grade.

In addition to this, I looked at the course material and asked "Is there anything thing on here that a doctor really needs to know?". I decided it was good for doctors to know how half-lives work, since this is important for things like drug dosing, as well as probably other things I don't even know about (since anything who's rate of decay is proportional to it's value will behave the same way). So, I explained to my students that a discharging capacitor was mathematically identical to the way that some drug concentrations decrease over time, and that there absolutely, positively would be a question about it on the exam. I didn't say anything else very specific about what would be on the rest of the exam. That exam had one question about a discharging capacitor, followed by a second question that was the same as the first, but reworked in terms of drugs. Most students got the first one right, but fewer got the second.

I think that part of my distaste, as an instructor, for students knowing a lot more about what is on the exam was that I wound up talking a lot more about the same things, and it got boring.

Exams and Overfitting

by robot-dreams 1 min read6th Jan 201547 comments


When I hear something like "What's going to be on the exam?", part of me gets indignant.  WHAT?!?!  You're defeating the whole point of the exam!  You're committing the Deadly Sin of Overfitting!

Let me step back and explain my view of exams.

When I take a class, my goal is to learn the material.  Exams are a way to answer the question, "How well did I learn the material?"[1].  But exams are only a few hours long, so it's unfeasible to have questions on all of the material.  To deal with this time constraint, an exam takes a random sample of the material and gives me a "statistical" rather than "perfect" answer to the question, "How well did I learn the material?"

If I know in advance what topics will be covered on the exam, and if I then prepare for the exam by learning only those topics, then I am screwing up this whole process.  By doing very well on the exam, I get the information, "Congratulations!  You learned the material covered on the exam very well."  But who knows how well I learned the material covered in class as a whole?  This is a textbook case of overfitting.

To be clear, I don't necessarily lose respect for someone who asks, "What's going to be on the exam?".  I understand that different people have different priorities[2], and that's fine by me.  But if you're taking a class because you truly want to learn the material, in spite of any sacrifices that you might have to make to do so[3], then I'd like to encourage you not to "study for the test".  I'd like to encourage you not to overfit.

[1] When I say "learned", I mean in the "Feynman" sense, not in the "teacher's password" sense.  I believe that a necessary (but not sufficient) condition for an exam to check for this kind of learning is to have problems that I've never seen before.

[2] Someone might care much more about getting into medical school than, say, mastering classical mechanics.  I respect that choice, and I acknowledge that someone might be in a system where getting a good grade in physics is required for getting into medical school, even though mastering classical mechanics isn't required for becoming a good doctor.

[3] There were a few terms when I felt like I did a really good job of learning the material (conveniently, I also got really good grades during these terms).  But for these terms, one (or both) of the following would happen:

  • I would take a huge hit in social status, because I was taking barely more than the minimum courseload.  At my university, there was a lot of social pressure to always take the maximum courseload (or petition to exceed the maximum courseload), and still participate in lots of extracurricular activities.
  • My girlfriend at the time would break up with me because of all the time I was spending on my coursework (and not with her).