Thanks for writing this! On the point of how to get information, mentors themselves seem like they should also be able to say a lot of useful things (though especially for more subjective points, I would put more weight on what previous mentees say!)
So since I'm going to be mentoring for MATS and for CHAI internships, I'll list my best guesses as to how working with me will be like, maybe this helps someone decide:
If you have concrete questions about other things, whose answer would make a difference for whether you want to apply, then definitely feel free to ask!
Strong +1 to asking the mentor being a great way to get information! My guess is many mentors aren't going out of their way to volunteer this kind of info, but will share it if asked. Especially if they've already decided that they want to work with you.
My MATS admission doc has some info on that for me, though I can give more detailed answers if anyone emails me with specific questions.
My thoughts from more or less the other side:
I think the average undergrad / new grad student (especially my past self) overvalues hard ambitious projects and undervalues easy straightforward projects. A good easy project should teach you lots of things that will be applicable in the future, and should contribute to the field at least a little bit, but should otherwise be almost as easy as possible.
To pick one example among many, "replicate some other paper but do better analyses on what happens as you tweak the architecture" is a huge family of easy, unsexy projects that will rapidly catapult you to being a world expert on a subject. I think if a mentor is suggesting easy projects that will teach you useful stuff, that should be a quite positive sign, not a reason for an "oh that's boring" reaction. And if you look at those easy projects and think "but the things it would teach me aren't the things I want to learn," try to think of an easy project that would teach you the things you want to learn.
On ranking mentors, I think people have good instincts about trading off expertise as a researcher versus goodness as a mentor, but they probably overvalue connections and status. If that status didn't come from banger research, you should generally ignore it.
I'm not sure I agree -- I think historically I made the opposite mistake, and from a rough guess the average new grad student at top CS programs tends to look too much for straightforward new projects (in part because you needed to have a paper in undergrad to get in, and therefore have probably done a project that was pretty straightforward and timeboxed).
I do think many early SERI MATS mentees did make the mistake you describe though, so maybe amongst people who are reading this post, the average person considering mentorship (who is not the average grad student) would indeed make your mistake?
Yep, I think the Law of Equal and Opposite Advice applies here.
One piece of advice which is pretty robust is — You should be about to explain your project to any other MATS mentee/mentor in about 3 minutes, along with the background context, motivation, theory of impact, success criteria, etc. If the inferential distance from the average MATS mentee/mentor exceed 3 minutes, then your project is probably either too vague or too esoteric.
(I say this as someone who should have followed this advice more strictly.)
I'd guess this varies by field? I think this would be bad advice in mech interp - there's a lot of concepts and existing mech interp theory that you need to understand a bunch of good projects, and people new to the field are often bad at explaining these (and, importantly, I think I have decent judgement about whether a project is any good). But I'd guess this is decent advice in some areas of alignment.
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
Several programmes right now offer people some kind of mentor or supervisor for a few months of research. I participated in SERI MATS 4.0 over the summer, and I saw just how different people's experiences were of being mentored. So this is my list of dimensions where I think mentors at these programmes can differ a lot, and where the differences can really affect people's experiences.
When you pick a mentor, you are effectively trading between these dimensions. It's good to know which ones you care about, so that you can make sensible tradeoffs.
Some mentors are mostly looking for research engineers to implement experiments for them. Others are looking for something a bit like research assistants to help them develop their agendas. Others are looking for proto-independent researchers/research leads who can come up with their own useful lines of research in the mentor's area.
I saw some people waver at the start of the programme because they expected their mentors to give them more direction. In fact, their mentors wanted them to find their own direction, and mentors varied in how clearly they communicated this. Conversely, I got the sense that some people were basically handed a project to work on when they would have liked more autonomy. So ask yourself: how much responsibility do you want to take for what you're working on? How much do you care about learning to do that?
I think this relates to seniority: my rough impression was that the most junior mentors were more often looking for something like collaborators to help develop their research, while more senior ones with more developed agendas tended to either want people who could execute on experiments for them, or want people who could find their own things to work on. But this isn't an absolute rule.
Engagement: Some mentors came into the office regularly. Others almost never did, even though they were in the Bay Area. Concretely, I think even though my team had a mentor on another continent, we weren't in the bottom quartile of mentorship time.
Nature of Engagement: It's not just how much time they'll specifically set aside to speak to you. How willing are they to read over a document and leave comments? How responsive are they to messages, and how much detail do you get? Also, some mentors work in groups, or have assistants.
Remoteness: Remoteness definitely makes things harder. You get a little extra friction in all conversations with your mentor, for starters. It's trickier to ever have really open-ended discussion with them. It's also easier to be a bit less open about your difficulties - if they can't ever look in your office then they can't see if you're not making progress, and it is very natural to want to hide problems. Personally, I wish we'd realised sooner that we had more scope for treating our mentor as more of a collaborator and less of a boss we needed to send reports to, and I think being remote made this harder.
A caveat here is that you can still talk to other mentors and researchers in person, which substitutes for some of the issues. But it is obviously not quite the same.
If you're an applicant anxiously wondering whether you'll even be accepted, it can be hard to notice that your mentor is an actual real human with their own personality. They will have been selected far more for their research than for their mentoring. So naturally different mentors will actually have very different personalities, strengths, and weaknesses.
Supportiveness: Some mentors will be more supportive and positive in general. Others might not offer praise so often, and it might feel more disheartening to work with them. And some mentees are fine without praise, but others really benefit from mentor encouragement.
High Standards: Some mentors are more laid back, others will have higher standards for progress, success, and work. This is really several dimensions, but definitely some streams had mentees staying late into the evening more often, some had a greater expectation that the research would lead to publication, and some were much more casual. Notably, mentors might have high standards in different areas.
Directness of Feedback: Some mentors will be very happy or even proactive about telling you exactly what they think you're doing wrong, or what you should be prioritising. I don't think any will refuse to give their opinions, but it's important to know how much you can assume that your mentor would have told you their concerns.
Weirdness: It's hard to elaborate on this, but you know it when you see it, and people often don't try to conceal it. Weirdness tends to bleed outwards. Some people actively like this, some are indifferent, and some are really switched off.
Expertise: Since we're listing tradeoffs, I do think it's worth taking the time to form your own opinions about the quality of your mentor's work, and what kinds of domains you can expect to get helpful advice from them on.
Mentoring/Management Experience: Mentors will vary a lot in how much experience they can bring to bear on mentorship itself. Some will have managed groups of researchers, but not all. And the kinds of relevant experience they have will probably inform what kind of mentorship they give.
Connections: Different mentors will have different networks which they can connect you to. This is particularly relevant if you need specific expertise from quite senior researchers.
This is a slightly awkward point, but an important one. There is a specific social scene in Berkeley which you might well find yourself joining if you stay there for a while as a research scholar. It has parties which can get wild, friendships and romances and longstanding grievances. Some mentors are in this social scene, or adjacent to it. And relatedly, I get the impression that some mentors are better at maintaining professional boundaries both in light of the above and otherwise.
Tricky to say. The best route is asking previous mentees, or other people that have worked with the mentor in the past: my guess is that most would be happy to reply to a short personal message that asked specific questions. Failing that, people's vague impressions from meeting someone and hearing things second-hand are pretty noisy, but they are still a useful signal for comparison. And I think you can get some sense from looking at the public profiles of mentors - their posting and comment histories, the kinds of work they've pursued, and how they present themselves.
Although the above can read a bit like a laundry list of potential problems, I do think these programmes are pretty good, and a fairly strict improvement over just trying to do research on your own. So best of luck!
Thanks to Henry Sleight and Sami Petersen for feedback on a draft, and Rohin Shah for suggesting I write something like this.