You missed one type of value, which is peer group. Most of the more difficult courses I've taken have given me at least some value by granting better access to more smart, competent people. This has a number of benefits:
Better human capital. By watching how more competent people work and think, you can often pick up useful study habits and better techniques for the subject you're studying. I've found this especially true in CS classes, where I've had this experience from both sides, e.g. teaching classmates how to use Git and picking up C coding style and tricks from better programmers.
Advice/planning help: both more advanced students and instructors can be very useful for the academic advice they provide later. Knowing talented students has given me info about several excellent courses, as well as summer opportunities, I wouldn't otherwise have known about. A professor who can become a good mentor is also invaluable (although if you don't plan to go into academia, you may have trouble finding a professor who can sympathize with your goals enough to give you good advice).
Better networking later: this is hopefully obvious, but knowing smart, competent people is useful for pretty much everything ever.
Friends/fun: people who think fast and have good background are less predictable/more interesting to talk to.
This suggests the following advice:
Seek out classes that signal difficulty or are known to be challenging. Many colleges have an accelerated introductory math sequence that will be good for this; in computer science, courses on operating systems and compilers are often known for their difficulty and attract the best/most interested students.
Avoid courses that are hard requirements for anything. These courses are the worst at filtering for good students because they're required. Additionally, they'll probably be taught at a low enough level that the good students will be disengaged and hard to spot.
Among sets of required alternatives, look for smaller courses. The largest course is probably the lowest common denominator. This is particularly important for general-education courses, in my experience.
Thanks, this is a great comment. A lot of these are points that I hadn't considered in depth, though some are points that I deliberately decided against including. That's because the focus of this post was about selecting between instructors who are offering different sections of what, on paper, is the same course. The questions of whether to take honors classes or what course level to do are questions I plan to explore in depth in future posts.
I did mention peer groups, but relegated that mention mostly to the PS of my post. That's because, in the particular narrow context of my post, I don't think there is a huge peer selection effect. What I mean is that different sections of what is nominally the same course will generally see fairly similar student populations, despite differences in the actual quality of the instructor or the instruction. There's two broad reasons: (i) most students aren't optimizing too much for instructor within sections at the same course level, but rather, are affected by scheduling considerations, and (ii) departments often impose space constraints on class sizes in such a manner as to make sure that each section has a similar size. Within these space limits, treatment tends to be first-come-first-serve. Thus, even if better peers want to take a class with a specific instructor, in the highly space-constrained and first-come-first-serve world, they're unlikely to be able to reliably succeed at doing so.
The case where peer selection effects would be strongest in such a first-come-first-serve enrollment process would be where an instructor develops a good reputation with students at a certain quality level and a bad reputation with students at lower quality levels, so that those students are actually trying to avoid the instructor.
Oh, sorry. I think I interpreted "assuming that the major and course level are given" differently than you intended. My university is too small to offer different sections of the same course by different instructors.
Better networking later: this is hopefully obvious, but knowing smart, competent people is useful for pretty much everything ever.
I'd actually like to see this better explored/supported... I believe it's true, and I've experienced it myself, but maybe if we could break down the specific ways in which networking can be useful, we could do it in a more optimized way (although if we optimize too hard, I assume we run the risk of coming across as phony?)
Avoid courses that are hard requirements for anything.
On the other hand, if you're not sure what you want to major in, you might try taking lots of these sort of classes to learn what you liked, start on long chains of prerequisites early, and keep your options open.
I'd actually like to see this better explored/supported... I believe it's true, and I've experienced it myself, but maybe if we could break down the specific ways in which networking can be useful, we could do it in a more optimized way (although if we optimize too hard, I assume we run the risk of coming across as phony?)
Collaboration: e.g., if I start a software company I'll be very happy that I know so many good programmers personally.
Signalling: for instance, people assume I'm more of a hardcore math person than I am because many of my roommates did well in math competitions.
Halo: I'm reasonably well-networked in some communities and people there seem to give my opinions a surprising amount of weight. Not sure if this is causation but it seems probable.
Favors: This has been the most important for me so far. Knowing people (who want to do things for you) is an almost fully general substitute for having skills. I'm not very good at biology, but my roommate knows enough to explain anything I don't understand. I'm bad at promoting my organization's events, but I can ask my friends on the student newspaper to help.
With regard to coming off as phony, I think in some circles there may be an implicit understanding that "networking" relationships are based on mutual benefit. YMMV though.
Instructors differ significantly from one another in terms of their quality of teaching (human capital + consumption value) and ease of grading (signaling). I'm curious here about the extent to which LessWrong readers think variation between instructors for similar courses compares to variation between institutions. For instance, how much does variation between instructors at Harvard compare with variation between UCLA and Harvard?
I had a few great professors (e.g. this guy), and mostly good ones, at De Anza College, a community college in Cupertino, California, to the point where I was a bit disappointed after transferring to UC Berkeley.
I had an Organic Chemistry teacher at the small community college in my home town that blew away my big university's Medicinal Chemistry prof. The big university prof was tenured & could treat his students as abusively as he desired without any fear of significant consequences.
I think there's a 4th type of value that students derive, implicitly, from courses that involve physical participation, especially: personal connections. Often these last well beyond the class, and many prominent businesses have been built by people who met in college.
Instructor aspects that would predict a student body likely to be a good pool of connections include assigning group tasks, encouraging peer-to-peer communication (e.g. debates, playing devil's advocate), organizing field trips, and borrowing from the corporate world: team-building exercises and outings in general.
It's quite rare for students to attend sample classes with instructors that they plan to study with
Every school I've been at had a strong tradition of a "shopping period" at the start of term where students tried out different classes, including different sections of the same class. Is this what you don't see happening? If this is something that varies a lot from school to school, it's probably just a matter of inertia, not of students having better methods. You suggest that lack of flexibility may discourage students from doing this, but my impression is that large inflexible state schools force a different kind of shopping through wait lists.
I've never seen it, but it might be better to sample a class in the middle of the previous term, to get a more representative sample; the drawback is that it might be incomprehensible.
@Douglas: " but it might be better to sample a class in the middle of the previous term" -- that's actually what I meant. I'll update the post to clarify. At UChicago, classes generally get full before the quarter begins, so it's hard to make changes even if one wants to, after the quarter begins. So people who don't sample classes the previous quarter are often stuck.
Also, thanks for sharing your thoughts! I really appreciate it.
Huh, this is a very different perspective than I've had at the U of C (Going into my third year undergraduate)
I've found that people chose a lot based on instructors. Even in my majors, chem and physics, where there's almost no electives, people often defer taking courses for a year when there's a better instructor.
As for evaluations, I can't tell you what most other students do. But when talking about the evaluations what my friends to decide whether or not to take a class, most of us don't pay much attention to the specific numbers they got, and focus on whether in the comments section people say the instructor knows what they're talking about, is interesting, and has fair tests.
As a sidenote, saying U of X for every X isn't a very specific determiner, and it appears you were attempting to be specific. For example, University of Calgary is the top-rated result for "U of C" on google, but you could easily be referring to University of Chicago or University of Connecticut, among other U of C institutions.
In context, it's almost certainly University of Chicago, given that this was the U of C mentioned in the post.
Yes, that's a good point. I had forgotten about that mention by the time I got to the comments.
I assumed it was obvious here, because the OP said he was talking about the University of Chicago.
@falenas108, thanks for sharing your thoughts. I genuinely appreciate it.
I've found that people chose a lot based on instructors.
Yes, I agree. I think some of my post carried the connotation that they don't, but that was unintended. I wrote:
Students routinely neglect instructor selection or use suboptimal criteria, relative to its importance. [...] Often, students decide their courses for the next semester or quarter after a meeting of a few hours with an advisor, based on time scheduling constraints.
So, to clarify, although I think it "often" happens that people select instructors randomly, it doesn't always happen. My impression is that there's heavy availability bias: if students can easily access evaluations, or know friends who have taken classes with a particular instructor, they'll use that data. If not, they'll select randomly. In general, my impression is that students are risk-averse and satisfice on this count: as long as an instructor seems "good enough" they won't generally try to look for a better one (this is particularly true for people in multi-course sequences with an instructor, we could call this a "status quo bias" or an "endowment effect" depending on your perspective). More proactive approaches, such as sitting in on classes that the instructors are teaching in the previous term, seem to be relatively rare.
My other point is that even when students are making proactive choices, the criteria they use may be suboptimal. I am not aware of high quality advice that would help students select instructors effectively. This is, I believe, in contrast with the vast (though possibly not very high quality) literature available on how to select a college or a major. If intra-institutional variation in instructor quality is comparable to inter-institutional variation, then there are probably unrealized gains in selecting instructors according to better criteria.
As for evaluations, I can't tell you what most other students do. But when talking about the evaluations what my friends to decide whether or not to take a class, most of us don't pay much attention to the specific numbers they got, and focus on whether in the comments section people say the instructor knows what they're talking about, is interesting, and has fair tests.
I've added an update to the post in response to this.
One thing I would note is that good teachers are rare and not very "good," but bad teachers are common and very very bad. This ties into Trevor's post: once you've avoided the bad teachers, it's probably not worth the much larger effort required to find an extra-bonus-good teacher that may or may not exist.
@linkhyrule5: Thanks for your thoughts, I really appreciate it.
I'd be grateful if you could provide some context regarding the reference class of educational environments for which you've made this statement (that the main problem is avoiding really bad teachers, rather than looking for really good ones).
I should probably have qualified my earlier post, as the only data I have to draw on is my own anecdotal evidence.
Nevertheless, with the qualifications that I went to a private high school and am currently at an Ivy college:
Sample size: since freshman high school, 26 "teachers" and nine "professors."
Definitions: a teacher/professor is good-I if I was particularly interested in his/her course (and not just the subject), and good-R if I retained particularly more from that course. A teacher/professor is bad-I if I was particularly uninterested in his/her course (and not just the subject), and bad-R if I retained particularly little from that course.
Of my teachers, 2 good-I teachers and 0 good-R teachers, with two "maybes" - not particularly good but above-average. 3 teachers who were both bad-I and bad-R - all of these are extreme cases, in which I learned almost nothing and loathed the class.
Of my professors, 0 good-I and no data on good-R (I'm a sophomore), but already 2 bad-I and I highly suspect bad-R.
A good instructor will often offer a good class that good students can take advantage of. A bad instructor will never offer a good class and no one can take advantage of it. Avoiding bad teachers is therefore always a move in a good direction, while selecting good teachers is only sometimes a move in a good direction.
I've been working in K-12 and college since 2005 as a sign language interpreter. The qualities of good instructors varies, but the qualities of bad instructors are shared. Late, unprepared, digressive without profit, argumentative, close minded, uses the class as political platform, uses the class as therapy session, uses the class as amateur comedy, unable / unwilling to earn respect (grudging or otherwise) of class - those are some of the things bad instructors share. Avoid those and you're more likely to succeed in any class.
A list of what to avoid is not anywhere near as useful as advice on how to identify those characteristics before the cost of not taking that course is too high (e.g. add/drop deadlines, full classes)
Another major factor for grads or advanced undergrads is the research that the professor is doing. This primarily comes into play during office hours (which are generally empty except before exams). Especially for established figures, this can be the only chance you'll have to get to know them (most are far too busy to take casual callers).
Even the worst lecturers are sometimes extraordinary one-on-one and, even when not, people doing interesting work tend to have far more than average contacts with other people doing interesting work. Show them that you're engaged and curious and invisible doors will open to you (many of the most interesting positions are never posted, but instead filled by qualified recommendations).
Data from my experience at Harvard: variability between instructors is very high (probably because many of them got professorships on the basis of their research rather than teaching; the two correlate but with a lot of noise). I'd guess this variance is much greater than institutional variance, though obviously I can't say for sure.
Also, your phrasing suggested that you thought Harvard had very high-quality teaching. It's probably good, but I wouldn't be surprised if most non-research colleges had higher average teaching quality, at least for college-level courses that don't require deep knowledge of highly advanced material. I'd guess that the advantage of taking classes at Harvard is more in the peer group (see other comment), and maybe the advanced undergrad or grad-level courses where command of the subject matter might vary more. (However, I've only ever taken classes at Harvard, so someone who's transferred in or out would have better info.)
Also, students here weigh instructor quite heavily in their course decisions, at least in the math department where professors switch courses almost every year.
Going into 3rd year undergrad at UChicago, I've selected all but one of my classes based on having been in at least a few class sessions with the same instructor. (The exception has extremely positive course evals.) I think being in a class with an instructor is the only valuable, reliable, consistently available source of information here - since students vary so much in learning style, learning ability, and background, it's hard to get good info from them. I think students don't shop as much as they could because that requires doing stuff.
The only times I pay attention to course evals is when they are mostly unanimous, or when they give specific indications of qualities that are definitely good or definitely bad. E.g. good: "gives lots of examples quickly, without being repetitive", "the homeworks are enlightening/challenging/not tedious" and bad: "doesn't speak clearly", "gives obscure/tedious homework problems".
By the way, Vipul, I think I have your old Linear Programming book from the Eckhart table. :)
Going off of what others have said, I'll add another reason people might satisfice with teachers.
In my experience, people agree much more about which teachers are bad than about which are good. Many of my favorite (in the sense that I learned a lot easily) teachers were disliked by other people, but almost all of those I thought were bad were widely thought of as bad. If you're not as interested in serious learning this might be less important.
So avoiding bad teachers requires a relatively small amount of information, but finding a teacher that is not just good, but good for you requires a much larger amount. So people reasonably only do the first part.
For math courses, I always took the "honors" version. This was not because I was an over-achiever but because the honors versions were taught by professors whereas the normal versions were taught be TAs. The instruction was soooooooooooo much better, which made up for the extra work and/or extra difficulty. My annoyance level was kept low, which was worth it.
@falenas108, thanks for sharing your thoughts. I genuinely appreciate it.
I've found that people chose a lot based on instructors.
Yes, I agree. I think some of my post carried the connotation that they don't, but that was unintended. I wrote:
Students routinely neglect instructor selection or use suboptimal criteria, relative to its importance. [...] Often, students decide their courses for the next semester or quarter after a meeting of a few hours with an advisor, based on time scheduling constraints.
So, to clarify, although I think it "often" happens that people select instructors randomly, it doesn't always happen. My impression is that there's heavy availability bias: if students can easily access evaluations, or know friends who have taken classes with a particular instructor, they'll use that data. If not, they'll select randomly. In general, my impression is that students are risk-averse and satisfice on this count: as long as an instructor seems "good enough" they won't generally try to look for a better one (this is particularly true for people in multi-course sequences with an instructor, we could call this a "status quo bias" or an "endowment effect" depending on your perspective). More proactive approaches, such as sitting in on classes that the instructors are teaching in the previous term, seem to be relatively rare.
My other point is that even when students are making proactive choices, the criteria they use may be suboptimal. I am not aware of high quality advice that would help students select instructors effectively. This is, I believe, in contrast with the vast (though possibly not very high quality) literature available on how to select a college or a major. If intra-institutional variation in instructor quality is comparable to inter-institutional variation, then there are probably unrealized gains in selecting instructors according to better criteria.
As for evaluations, I can't tell you what most other students do. But when talking about the evaluations what my friends to decide whether or not to take a class, most of us don't pay much attention to the specific numbers they got, and focus on whether in the comments section people say the instructor knows what they're talking about, is interesting, and has fair tests.
I've added an update to the post in response to this.
One thing I would note is that good teachers are rare and not very "good," but bad teachers are common and very very bad. This ties into Trevor's post: once you've avoided the bad teachers, it's probably not worth the much larger effort required to find an extra-bonus-good teacher that may or may not exist.
I've been trying to compile advice on how to select good instructors in college, assuming that the major and course level are given. Of course, many people don't have a lot of flexibility with instructor choice, so this advice has limited applicability. The reason I'm posting to LessWrong is to get feedback from interested LessWrong readers regarding some of the factual assertions I make. I've marked the parts where I'm most eager for feedback. My experience with instruction (direct and indirect) has been focused largely on mathematics. Thus, it's possible that some of the points I raise have limited applicability in more discussion-based subjects.
Note also that although my experiences have been partly shaped by the undergraduate teaching I've done in four years as a graduate student at the University of Chicago, I'm neither representing the University nor am I critiquing or describing any features specific to the University of Chicago.
I'm working within the conceptual framework that there are three broad types of value that students derive from college courses:
Instructors differ significantly from one another in terms of their quality of teaching (human capital + consumption value) and ease of grading (signaling). I'm curious here about the extent to which LessWrong readers think variation between instructors for similar courses compares to variation between institutions. For instance, how much does variation between instructors at Harvard compare with variation between UCLA and Harvard?
Apart from the teaching and grading, a good instructor is a candidate to write recommendation letters for future courses, internships and scholarships, and graduate school applications. Additionally, a good instructor may recommend options to students.
Students routinely neglect instructor selection or use suboptimal criteria, relative to its importance (is this true?) or relative to other things they fret over, such as college selection. Often, students decide their courses for the next semester or quarter after a meeting of a few hours with an advisor, based on time scheduling constraints. Update: Other students use data such as online evaluations and talking to friends who have taken classes with specific instructors, but may not take proactive steps to collect such data. Further, the criteria that they use to process the data may well be suboptimal (or at any rate, I believe so, which is why I'm writing the post).
For the discussion below, I largely assume that the student is selecting between different sections of the same course, where each section follows a similar curriculum but the instructor has flexibility in terms of the final examinations and grades. What do LessWrong readers think about how common this is relative to a setup where all sections have the same final examinations and grades? However, much of the advice is general.
Students often incorrectly believe that instructors who are harder in terms of the material covered or in terms of the work they assign will be harder in terms of the final letter grades they assign. This is not necessarily true. The overall correlation sign is unclear, but there are teachers in all four quadrants in terms of (hardness of material, hardness of grading).
Looking at the difficulty level of examinations in isolation can paint a misleading picture, because instructors may differ somewhat in the techniques they cover, and this can radically affect the perceived difficulty of a question.
Remember that student evaluations, although more reliable than peer faculty evaluations or other evaluation methods, are still generally unreliable (the Wikipedia page and Mike Huemer's article are great starting points). There is still some value you can extract from student evaluations, but not a lot. The main problem with the numerical part of such evaluations is improper anchoring – students aren't clear on the scale relative to which they are evaluating instructors. As a result, the median numerical rating winds up at around 4.4/5, leading to censoring at the top (a ceiling effect). Update: Since Falenas108 raised this issue in a comment, I know that the long form responses in student evaluations are somewhat more informative than the numerical ratings, but these suffer from many of the same problems. The Huemer article and the references in the Wikipedia article and the Huemer article are good starting points. Further, optional online evaluations (which are the only ones that are easily available) often suffer from both low response rates and a selection bias in the set of respondents towards people who feel strongly about the instructor.
The illusion of transparency and double illusion of transparency (see also here) make it quite hard to evaluate teachers by just sampling their classes. It does provide a start though, and, if done well, can be more informative than student evaluations.
Criterion-referenced versus norm-referenced grading: Keep in mind the distinction between criterion-referenced grading (assigning grades based on attainment of pre-specified mastery levels) and norm-referenced grading (assigning grades based on relative performance). One extreme of norm-referenced grading is “grading on a curve” where it is pre-specified how many students will get an A, how many will get an A-, etc. There are also other forms of norm-referenced grading (a typical approach many instructors use is to start from the top and “look for a gap” of x or more points for each grade decrement). Note that a lot of students confuse “grading on a curve” with “having lax grading standards” and conflate both of these with “having low numerical cutoffs for a given grade” but the three ideas are all different (particularly in the college course context) and it's best that you do not conflate them.
The significance of measurement error: There are three factors that affect the degree of measurement error in grading that you can easily determine:
The degree of gap between the score requirements for grades. An easy test where a 90 is an A, an 85 is an A-, etc., means that one careless error can affect your grade adversely. A hard test where an 80 is an A, a 60 is an A-, etc., means that the measurement is considerably less likely to be influenced by small measurement error issues.
The number of questions on the test, and the number of tests and other assessment methods used. Generally, the more the number of items used, and the more distinct questions on the test, the more reliable the measurement (this is something to do with the law of large numbers).
Whether or not instructors award partial credit in tests. Partial credit makes measurement more accurate, by indirectly increasing the number of items being measured.
Teaching to the test: Instructors differ significantly in the extent to which they teach to the test. Keep in mind that “teaching to the test” although often frowned upon from a human capital perspective is not necessarily bad from either a human capital or a signaling perspective. That's because teaching to the test can be thought of as “testing what is taught” particularly when the instructor has flexibility in setting the test. Degrees of teaching to the test include:
Providing sample tests to eliminate uncertainty about the test format.
Providing review materials or conducting review sessions that are closely optimized to maximal test performance.
In general, if you are a good student with a reasonable shot at getting a fairly good grade, and you are interested in both human capital and grades, give preference to a criterion-referenced instructor who boasts low measurement error (lots of questions, lots of tests, partial credit, and hard tests) and teaches to the test. Subject to these constraints, favor the instructor whose overall grades are easiest/best.
If you are not interested in the human capital component of the course much, and are looking purely for a good grade, the choice between criterion-referenced and norm-referenced grading is harder. If choosing a norm-referenced grader, keep in mind the student population you are with. Depending on other courses, the student body during some quarters or semesters can be considerably better than during others, making it relatively harder to get a good grade.
If you are not very good with the material and not keen on deep learning, it might actually pay off to choose an instructor with higher measurement error. This is similar to the idea that people with the odds against them would have the most chance of winning if they made a small number of big bets, whereas people with the odds in their favor would have the most chance of winning by placing a large number of small bets.
It's quite rare for students to attend sample classes with instructors that they plan to study with (there could be many reasons -- do readers have any thoughts?). Update: When talking of "sample classes" I was referring to classes with the same instructor in an earlier term, because it's often very difficult to change classes once the term has begun, due to complications with scheduling, or classes getting full. The same instructor may be teaching a somewhat different course in the preceding term, so such sampling is useful only in so far as it captures generic aspects of instructors that transfer across courses. There may also be an element of difference between universities with a semester system and a quarter system. Those on the quarter system have less time to shop around between classes because the schedule is more compressed overall.
For those who do choose to attend sample classes, the following tips may be useful.
If you attend sample classes, check for how carefully students are listening: When attending a sample class, take a seat close to the back, so you get a bird's eye view of what all the students are doing. Existing students have more experience than you and are more tuned in to the overall course, so their decision of how much attention to pay to the teacher is an indication of the value generated by the teacher.
Judge favorably instructors who engage in cold calling and non-voluntary participation (polling, desk work checking). Ceteris paribus, if shown two equally “interactive” classes, one with cold calling and one without, pick the one with cold calling. In general, pick classes with cold calling. Polling and desk work are also positives, albeit milder ones, and many college-level classes don't have much time for desk work.
Avoid classes where students appear eager to impress the teacher or their fellow students: A dynamic where students are participating largely with the goal of impressing the teacher or their fellow students tends to be unhealthy in promoting student learning.
Favor good classroom technique, but don't be too impressed by slickness due to double illusion of transparency. Good board technique, neat handwriting, etc. are valuable. However, sometimes better instructors actually sound more confusing than not-so-good instructors, because they highlight areas of difficulty, cold call students, etc., making the pain points clearer and avoiding the double illusion of transparency.
The following may be harder to gauge from a single lecture, but can usually be better gathered by talking to students who have studied with the instructor and in some cases by reviewing student evaluations that include long form responses.
Favor instructors who make notes and sources clear. This is less of an issue if the instructor is rigidly following a specific course text and the text is good. Insofar as the instructor is covering material not in the text, does the instructor provide notes or sources to read, or expect people to use notes taken in class?
Favor frequent “low-stakes” assessment to shatter the double illusion of transparency. Instructors who care about learning will tend to be more likely to use frequent low-stakes assessment such as class quizzes. These indicate that the students and instructor generally have a clear idea at any given stage of how well the material is being understood. However, beware of the difficulty level of these. Very easy assessments and very difficult assessments should be discounted (both of these have their uses, but they don't shatter the double illusion of transparency). Although the specifics vary, low-stakes assessments are best when the average score is somewhere between 30% and 70%.
When looking at student evaluations, discount evaluations that make it appear like the instructor “made everything very easy” – this is likely a double illusion of transparency or a student who had an unusually strong background. Many concepts being taught are hard. While it's possible to obfuscate them with difficult instruction, it's not possible to make them “very easy.” It's quite likely that instructors who give their students this impression are not explaining the concepts fully, or are testing the students in a sufficiently superficial manner that the students think that what they got just by hearing the material is a full understanding. It's also possible that some student was unusually smart or had an unusually strong background and committed a fundamental attribution error by attributing it to the instructor. Of course, good instructors do end up teaching their students better and perhaps even making (some of) the concepts completely clear, but rarely by making everything seem “very easy” – there is usually some sort of effort and pain undergone by the students to acquire that understanding.
When looking at student evaluations or getting word-of-mouth feedback, weigh more heavily certain kinds of evaluations that overcome the problem of bad anchoring and adjustment. Evaluations by people who have switched between multiple sections of the same course are often most illuminative, because they can control for the difficulty level of the material to quite an extent. Evaluations by people who have taken follow-on courses that rely on the material more are also useful because these people have more of an idea of whether what they learned was actually used, and can judge the quality of learning in that context.
Weigh positively (but don't be overimpressed by) student recalls of “aha” moments and phrases that indicate that students felt that the learning in their class was at a higher plane than in the others. “Aha” moments and unexpected connections emerging between seeds sown earlier in the course and material covered recently are indicative of unusually good teachers. However, keep in mind that given the huge subject matter knowledge gap between the teacher and the student, it is relatively easy for students to have false “aha” moments for insights that are actually pretty mundane and ones that they should have got at the outset.
(This one may be asking too much, or may be too idiosyncratic): Ask students to recall their test preparation experience. This can often be insightful regarding the kind of methods and level of learning the instructor has encouraged. Note that students differ, so this should be considered only averaged across students or for students for whom you know their general pattern of test preparation. Good instructors will try to maximize the extent to which test preparation is used to reinforce the material, rather than simply treating tests as a method to assess students. If instructors put up review materials or conduct review sessions, ask about the format of these. Negatively judge review sessions that are just somewhat tweaked versions of lectures. Look for review sessions that are “controlled, desk-work-based, and interactive” – controlled in the sense that the instructor controls the flow of the session. The “desk-work-based” says that the review session should largely involve desk work by all rather than lecturing or board work by select individuals. The “interactive” component stresses interaction between the instructor and students. In addition to the review sessions, look for review materials, and for students' subjective experience when reviewing. Did they have “aha” moments at review time? (Note that desk-work becomes more important at review time than cold-calling, because students are required to be able to work things out in full detail rather than just answer isolated questions).
What do readers think of the lists I've offered above? Are there items you disagree with? Things you think I missed? Arguments that the entire question is ill-considered? All feedback would be very much appreciated.
PS: I've deliberately omitted other factors, such as scheduling and space constraints and peer choices, from the discussion. In many cases, the fact that a close friend or study buddy is taking a particular section is good reason to take that section. Similarly, scheduling and space constraints can be binding at times. I don't think that these meaningfully alter the shape of the preceding advice, but they do constrain the scope within which it can be applied.