Last updated Sept 2 2025
TL;DR
- This post is about the mindset and process I recommend if you want to do mechanistic interpretability research. I aim to give a clear sense of direction, so give opinionated advice and concrete recommendations.
- Mech interp is high-leverage, impactful, and learnable on your own with short feedback loops and modest compute.
- Learn the minimum viable basics, then do research. Mech interp is an empirical science
- Three stages:
- Stage 1: Learning the Ropes
- Unpacking the research process:
- Many skills, categorise them by the feedback loops.
- Fast skills (minutes-hours) like write/run/debug experiments
- Slow (weeks) like how to prioritise and when to pivot
- Very slow (months) like generating good research ideas
- Do not try to learn all skills at once. Focus on fast/medium skills first, then slowly expand
- 4 phases of research: finding an idea (ideation) -> building intuition and hunches (exploration) -> testing hypotheses (understanding) -> refining and writing up (distillation)
- Stage 2: Mini projects (1-5 days each for 2-4 weeks)
- Exploration mindset: Maximise information gain per unit time, learn how to get unstuck. You don't need a plan, so long as you're learning
- Understanding mindset: Every research result is false until proven otherwise. The more exciting a result is, the more likely it is to be false. Be your own greatest critic
- Idea quality (ideation) and write-ups (distillation) aren't the priority yet; taste and prioritization are learned by doing things.
- Having good research ideas takes forever to learn, to choose early projects, cheat! Pick well scoped projects, eg extending a paper (ideas)
- Use LLMs extensively - they should speed up your research/coding a lot (if you know how to use them properly!)
- Stage 3: Towards full projects
- Work in 1-2 week sprints, post-mortem after each, pivot to another project unless it's going great
- Slower skills and key mindsets: careful skepticism, awareness of the literature, prioritization, high productivity
- Do good science, not flashy science - be honest about limitations, give proof you're not cherry picking, read your data, do the simple things that work, use real baselines.
- Write-up your work! Distill it into a narrative, then iteratively expand it to a write-up
- Practice generating research ideas. If possible, try to imitation learn a mentor's research taste.
- Proactively reach out to mentors Everything is much easier with a good mentor. Cold email, apply for mentoring programs, etc.
- Reach out to researchers who'll have time, not the most famous
- Careers: If you want to work in the field, apply for things! Jobs, mentoring programs, funding, academic labs.
- I also give various thoughts on how I'm thinking about the field nowadays, and what I’ve changed my mind about. I separate these from the practical advice, so you can take it or leave it.
Introduction
Mechanistic interpretability (mech interp) is, in my incredibly biased opinion, one of the most exciting research areas out there. We have these incredibly complex AI models that we don't understand, yet there are tantalizing signs of real structure inside them. Even partial understanding of this structure opens up a world of possibilities, yet is neglected by 99% of machine learning researchers. There’s so much to do!
I think mech interp is an unusually easy field to learn about on your own: there’s a lot of educational materials, you don’t need too much compute, and there’s short feedback loops. But if you're new, it can feel pretty intimidating to get started. This is my updated guide on how to skill up, get involved, reach the point where you can do actual research, and some advice on how to go from there to a career/academic role in the field.
This guide is deliberately highly opinionated. My goal is to convey a productive mindset and concrete steps that I think will work well, and give a sense of direction, rather than trying to give a fully broad overview or perfect advice. (And many of the links are to my own work because that's what I know best. Sorry!)
High-Level Framing
My core philosophy for getting into mech interp is this: learn the absolute minimal basics as quickly as possible, and then immediately transition to learning by doing research.
The goal is not to read every paper before you touch research. When doing research you'll notice gaps and go back to learn more. But being grounded in a project will give you vastly more direction to guide your learning, and contextualise why anything you’re learning actually matters. You just want enough grounding to start a project with some understanding of what you’re doing.
Don't stress about the research quality at first, or having the perfect project idea. Key skills, like research taste and the ability to prioritize, take time to develop. Gaining experience—even messy experience—will teach you the basics like how to run and interpret experiments, which in turn help you learn the high-level skills.
I break this down into three stages:
- Learning the ropes, where you work through the basics breadth first, and after at most a month, move on to stage 2
- Practicing research with mini-projects. Work on throwaway, 1-5 day research projects. Focus on practicing the basic research skills with the fastest feedback loops, don’t stress about having the best ideas, or writing them up. After 2-4 weeks, move on to stage 3
- Work up to full-projects: work in 1-2 week sprints. After each, do a post-mortem and pivot to something else, unless it was going great and has momentum. Eventually, you should end up working on something longer-term. Start thinking about the deeper skills and research mindsets, practice having good ideas, and prioritize making good public write-ups of sprints that went well
Stage 1: Learning the Ropes
Your goal here is learning the basics: how to write experiments with a mech interp library, understanding the key concepts, getting the lay of the land.
Your aim is learning enough that the rest of your learning can be done via doing research, not finishing learning. Prioritize ruthlessly. After max 1 month, move on to stage 2. I’ve flagged which parts of this I think are essential, vs just nice to have.
Do not just read papers - a common mistake among academic types is to spend months reading as many papers as they can get their hands on before writing code. Don’t do it. Mech interp is an empirical science, getting your hands dirty gives key context for your learning. Intersperse reading papers with doing coding tutorials or small research explorations. See my research walkthroughs for an idea of what tiny exploratory projects can look like.
LLMs are a key tool - see the section below for advice on using them well
Assuming you already know basic Python and introductory ML concepts.
- Maths:
- Linear Algebra is King (Essential): You need to think in vectors and matrices fluently. This is by far the highest value set of generic math you should learn to do mech interp or ML research.
- Resource: 3Blue1Brown's Essence of Linear Algebra.
- Highly recommended: Put A Mathematical Framework For Transformer Circuits in the context window and have the LLM generate exercises to test your intuitions about transformer internals.
- LLMs are great for checking whether linear algebra actually clicks. Try summarizing what you've learned and the links between different concepts and ask an LLM whether you are correct. For example:
- Ensure you understand SVD and why it works
- What does changing basis mean and why does it matter
- Key ways a low rank and full rank matrix differ
- Other Bits: Basic probability, info theory, optimization, vector calculus.
- Use an LLM tutor to quiz your understanding on the parts most relevant to transformers
- Generally don’t bother learning other areas of maths (unless doing it for fun!)
- Practical ML with PyTorch: (Essential)
- Cloud GPUs:
- You’ll need to be able to run language models, which (typically) needs a GPU
- You can start with Google Colab to get started fast, but it’ll be very constraining to use long-term. Learn to rent and use a cloud GPU.
- Newer Macbook Pros, or computers with powerful gaming GPUs may also be able to run LLMs locally
- Resource: ARENA has a guide. I like runpod.io as a provider; vast.ai is cheaper.
- nnsight also lets you do some interpretability on certain models they host themselves, including LLaMA 3 405B, which can be a great way to work with larger models.
Mechanistic Interpretability Techniques
A lot of mech interp research looks like knowing the right technique to apply and in what context. This is a key thing to prioritise getting your head around when starting out. You’ll learn this with a mix of reading educational materials and doing coding tutorials like ARENA (discussed in next sub-section).
Understanding the literature
Your priority is to understand the concepts and the basics, but you want a sense for the landscape of the field, so you should practice reading at least some papers.
- Remember, breadth over depth. Skim things, get a sense of what's out there, and only dive into the things that are most interesting.
- You should be heavily using LLMs here. Give them something you're considering reading and get a summary, ask questions about the work, summarise your understanding to it and ask for feedback (with an anti-sycophancy prompt).
- If you aren't able to verify yourself, cross-reference by asking multiple LLMs and making sure they all say consistent things.
- Here’s a list of my favourite papers (as of mid 2024) with summaries and opinions
- Open Problems In Mechanistic Interpretability is a decent recent literature review, that a lot of top mech interp people were involved in
- Be warned that the paper basically consists of a bunch of opinionated and disagreeable researchers writing their own sections and often having strong takes. Don’t defer to it too much, but it's a good way to quickly assess what's out there.
- Deep dives: You should read at least one paper carefully and in full. This is a useful skill that you will use in research projects where there’s a handful of extremely relevant papers to your project
- This is much more than just reading the words! You should write out a summary, try to understand the surrounding context with LLM help, be able to describe why the paper exists, the motivation, the problem it's trying to solve, etc.
- Aim for a barbell strategy: put minimal effort into most papers and a lot of effort into a few.
- LLMs: LLMs are a super useful tool for exploring the literature, but easy to shoot yourself in the foot with.
- As a search engine over the literature (especially with some lit reviews in context, or a starting paper), basically doing a lit review, finding relevant work for a question you have, etc.
- As a tool to help with deep dives - you need to actually read the paper, but I recommend having the LLM chat open as you read with the paper in the context and asking it questions, for context, etc every time you get confused.
Using LLMs for Learning
Note: I expect this section to go out of date fast! Written early Sept 2025
LLMs are a super useful tool for learning, especially in a new field. While they struggle to beat experts, they often beat novices. If you aren’t using them regularly throughout this process, I’d guess you’re leaving a bunch of value on the table.
But LLMs have weird flaws and strengths, and it’s worth being intentional about how you use them:
Interlude: What is mech interp?
Feel free to skip to the “what should I do next” part
At this point it’s worth reflecting on what mech interp actually is. What are we even doing here? There isn't a consensus definition on how exactly to define mechanistic interpretability, and different researchers will give very different takes. But my working definition is as follows.
- Interpretability is the study of understanding models, gaining insight into their behavior, the cognition inside of them, why and how they work, etc. This is the important part and the heart of the field.
- Mechanistic means using the internals of the model, the weights and activations
- So mechanistic interpretability is any approach to understanding the model that uses its internals.
- This is distinct from some other worthwhile directions, like black box interpretability, understanding models without using the internals, and model internals, using the internals of the model for other things like steering vectors.
Why this definition? To do impactful research, it's often good to find the directions that other people are missing. I think of most of machine learning as non-mechanistic non-interpretability. 99% of ML research just looks at the inputs and outputs to models, and treats its north star as controlling their behavior. Progress is defined by making a number go up, not to explain why it works. This has been very successful, but IMO leaves a lot of value on the table. Mechanistic interpretability is about doing better than this, and has achieved a bunch of cool stuff, like teaching grandmasters how to play chess better by interpreting AlphaZero.
Why care? Obviously, our goal is not “do things if and only if they fit the above definition”, but I find it a useful one. To discuss this, let’s first consider our actual goals here. To me, the ultimate goal is to make human-level AI systems (or beyond) safer. I do mech interp because I think we’ll find enough understanding of what happens inside a model to be pragmatically useful here (also, because mech interp is fun!): to better understand how they work, detect if they're lying to us, detect and diagnose unexpected failure modes, etc. But people’s goals vary, e.g. real-world usefulness today, aesthetic beauty, or scientific insight. It’s worth thinking about what yours are.
Some implications of this framing worth laying out:
- My ultimate north star is pragmatism - achieve enough understanding to be (reliably) useful. Subgoals like “completely reverse engineer the model” are just means to an end.
- One of my big shifts in research prioritization in recent years is concluding that reverse engineering is not the right aim. Instead, I think we should just more directly try to do pragmatic work that enables us to do useful things using internals. I discuss this shift more later on.
This is a broad definition. Historically, the field has focused on more specific agendas, like ambitious reverse engineering of models. But I think we shouldn’t limit ourselves, there’s many other important and neglected directions and the field is large enough to cover a lot of ground
- It’s about understanding, not just using internals - model internals methods like steering vectors can be useful for shaping a model’s behaviour, but compete with many powerful methods like prompting and fine-tuning. Very few areas of ML can achieve understanding
- Don’t be a purist - using internals is a means to an end. If black-box methods are the right tool, use them
The Big Picture: Learning the Craft of Research
So, you've gone through the tutorials, you understand the core concepts, and you can write some basic experimental code. Now comes the hard part: learning how to actually do mech interp research.
This is an inherently difficult thing to learn, of course. But IMO people often misunderstand what they need to do here, try to learn everything at once, or more generally make life unnecessarily hard for themselves. The key is to break the process down, understand the different skills involved, and focus on learning the pieces with the fastest feedback loops first.
I suggest breaking this down into two stages.
Stage 2: working on a bunch of throwaway mini projects of 1-5 days each. Don't stress about choosing the best projects or producing public output. The goal is to learn the skills with the fastest feedback loops.
Stage 3: After a few weeks of these, start to be more ambitious: paying more attention to how you choose your projects, gaining the subtler skills, and how to write things up. I still recommend working iteratively, in one to two week sprints, but ending up with longer-term projects if things go well.
Note: Unlike stage 1 to 2, the transition from stages two to three should be fairly gradual as you take on larger projects and become more ambitious. A good default would be after three to four weeks in stage two, but you don’t need to have a big formal shift.
Mentorship: A good mentor is a major accelerator, and finding one should be a major priority for you. In the careers section, I provide advice on how to go about finding a good mentor, and how concretely they can add value. In the rest of the post I'll write most of it assuming you do not have a mentor and then flag the ways to use a mentor where appropriate.
Unpacking the Research Process
I find it helpful to think of research as a cycle of four distinct stages. Read my blog post on the research proces for full details, but in brief:
- Ideation: You choose a research problem or a general domain to focus on.
- Exploration: You may not have a specific hypothesis yet; you’re just trying to figure out the right questions to ask, and build deeper intuition for the domain. Your north star is to gain information and surface area.
- Understanding: This begins when you have a concrete hypothesis, and some intuitive understanding of the domain. Your north star is to convince yourself that the hypothesis is true or false.
- Distillation: Once you’re convinced, your north star is to compress your findings into concise, rigorous truth that you can communicate to the world - create enough experimental evidence to convince others, write it up clearly, and share it.
Underpinning these stages is a host of skills, best separated by how quickly you can apply them and get feedback. We learn by doing things and getting feedback, so you’ll learn the fast ones much more quickly. I put a rough list and categorization below.
My general advice is to prioritize learning these in order of feedback loops. If it seems like you need a slow skill to get started, like the taste to choose a good research problem, find a way to cheat rather than stressing about not having that skill (e.g. doing an incremental extension to a paper, getting one from a mentor, etc).
- Fast Loop (minutes-hours):
- Planning and writing experiment code
- Medium: Designing great experiments
- Medium: Knowing when to write hacky vs. quality code.
- Running/debugging experiments
- Medium/Slow: Spotting and fixing subtle bugs (e.g., you got your tokenization subtly wrong, you didn’t search hyper-parameters well enough, etc)
- Interpreting the results of a single experiment.
- Medium: Understanding whether your results support your conclusions
- Slow: Spotting subtle interpretability illusions where your results don't actually support your claims
- Medium Loop (days):
- Developing a conceptual understanding of mech interp
- Slow: Noticing and fixing your own subtle confusions
- Slow: Build a deep knowledge of the literature
- Knowing how to explore without getting stuck
- Writing up results
- Slow: Communicating your work in a way that’s genuinely clear to people.
- Slow: Communicating why your work is interesting to people
- Slow Loop (weeks):
- Prioritizing which experiment to do next
- Knowing when to continue with a research direction or pivot to another angle of attack/another project
- Identifying bad research ideas, without doing a project on them first
- Very Slow Loop (months):
- Coming up with good research ideas. This is the core of "research taste."
Your progression should be simple: First, focus on the fast/medium skills behind exploration and understanding with throwaway projects. Then, graduate to end-to-end projects where you can intentionally practice the deeper skills, and practice ideation and distillation too.
What is research taste?
A particularly important and fuzzy type of skill is called research taste. I basically think of this as the bundle of intuitions you get with enough research experience that let you do things like come up with good ideas, predict if an idea is promising, have conviction in good research directions, etc. Check out my post on the topic for more thoughts.
I broadly think you should just ignore it for now, find ways to compensate for not having much yet, and focus on learning the fast-medium skills, and this will give you a much better base for learning it. In particular, it's much faster to learn with a mentor, so if you don't have a mentor at the start, you should prioritize other things.
But you want to learn it eventually, so it's good to be mindful of it throughout, and look for opportunities to practice and learn lessons. I recommend treating it as a nice-to-have but not stressing about it
Note, one important trap here is that having good taste can often manifest as having confidence and conviction in some research direction. But often novice researchers develop this confidence and conviction significantly before they develop the ability to not be confident in bad ideas. It’s often a good learning experience to once or twice pursue a thing you feel really convinced is going to be epic and then discover you're wrong, so it's not that bad an outcome, especially in stage 2 (mini-projects) but be warned.
Stage 2: Practicing Research with Mini-Projects
With that big picture in mind, let's get our hands dirty. You want to do a series of ~1-5 day mini-projects, for maybe 2-4 weeks. The goal right now is to learn the craft, not to produce groundbreaking research.
Focus on practicing exploration and understanding and gaining the fast/medium skills, leave aside ideation and distillation for now. If you produce something cool and want to write it up, great! But that’s a nice-to-have, not a priority.
Once you finish a mini-project, remember to do a post-mortem. Spend at least an hour analyzing: what did you do? What did you try? What worked? What didn't? What mistakes did you make? What would you do differently if doing this again? And how can you integrate this into your research strategy going forwards?
Choose A Project
Some suggested starter projects
- Replicate and Extend a Paper: A classic for a reason. Replicate a key result, then extend it. Suggestions:
- Refusal is mediated by a single direction
- Extending papers can vary a lot in difficulty. For example, applying the method to study refusal on a new model is easy as you can reuse the same data, while applying it to a new concept is harder.
- Skills: practicing activation patching and steering vectors.
- Thought Anchors: apply these reasoning model interpretability methods to new types of prompts, or explore some prompts using the linked interface, or see if you can improve on the methods/invent your own.
- Skills: reasoning model interpretability, using LLM APIs, and working with modern models
- Replicate the truth probes in Geometry of Truth on a more modern model and try applying them in more interesting settings. How well do they generalise? Can you break them? If so, can you fix this?
- Skills: probing, supervised learning, dataset creation
- Play around with something interesting:
Those cover two kinds of starter projects:
- Understanding-heavy, where you take a well-known domain and try to test a hypothesis there (e.g. extending a paper you’ve read closely)
- Note that you still want to do some
- Exploration-heavy, where you take some phenomena (a technique, a model, a phenomena, etc) play around with it, and try to understand what’s going on.
- Exploration-heavy projects are often a less familiar style, so make sure to do some of those!
Common mistakes:
- People often get hung up on finding the “best” project. Sadly, that’s not going to happen. Instead, just do something and see what happens - better ideas and inspiration come with time.
- Don't get too attached to your first project. It was probably badly chosen! These are throwaway projects, just move on once you’re not learning as much.
- Conversely, don't flit between ideas so much that you never build your "getting unstuck" toolkit.
- Avoid compute-heavy and/papers (e.g., training cross-layer transcoders) or highly technical papers (e.g., Sparse Feature Circuits).
Practicing Exploration
The idea of exploration as a phase in itself often trips up people new to mech interp. They feel like they always need to have a plan, a clear thing they're doing at any given point, etc. In my experience, you will often spend more than half of a project trying to figure out what the hell is happening and what you think your plan is. This is totally fine!
You don't need a plan. It's okay to be confused. However, this does not mean you should just screw around. Your North Star: gain information and surface area on the problem. Your job is to take actions that maximise information gained per unit time. If you've learned nothing in 2 hours, pivot to another approach. If 2-3 approaches were dead ends, it’s fine to just pick another problem.
I have several research walkthroughs on my YouTube channel that I think demonstrates the mindset of exploration. What I think is an appropriate speed to be moving. E.g. I think you should aim to make a new plot every few minutes (or faster!) if experiments don't take too long to run.
A common difficulty is feeling “stuck” and not knowing what to do. IMO, this is largely a skill issue. Here's my recommended toolkit when this happens:
- Use "gain surface area" techniques, things that can surface new ideas and connections and just give you raw data to work with: look at the model's output/chain-of-thought, change the prompt, probe for a concept, look at an SAE/attribution graph, read examples from your dataset, try logit lens or steering, etc.
- Set a 5-minute timer and brainstorm things you're curious about or directions to try.
- If you’re confused/curious about something, set a 5 minute timer and brainstorm what could be happening.
Other advice:
- Before any >30 minute experiment, stop and brainstorm alternatives. Is this really the fastest way to gain information?
- It's totally fine to pause for half a day to go learn some key background knowledge.
- Get in the habit of keeping a research log of your findings and a "highlights" doc for the really cool stuff.
- If applicable, it can be cool to have your research log be a slack/discord channel
- Remember: when exploring and thinking through how to explain mysterious phenomena, most of your probability mass should be on "something I haven't thought of yet."
- Practice following your curiosity, but be aware that it’ll often lead you astray at first. When it does, pay attention! What can you learn from this?
Practicing Understanding
If exploration goes well, you'll start to form hunches about the problem. E.g. thinking that you are successfully (linearly) probing for some concept. Or that you found a direction that mediates refusal. Or that days of the week are represented as a circle in a 2D subspace.
Once you have this, you want to go to figure out if it's actually true. Be warned, the feeling of “being really convinced that it's true” is very different from actually being true. Part of being a good researcher is being good enough at testing and falsifying your pet hypotheses that, when you fail to falsify one, there’s a good chance that it's true. But you're probably not there yet.
Note: While I find it helpful to think of these as discrete stages, often you'll be flitting back and forth. A great way to explore is coming up with guesses and micro-hypotheses about what's going on, running a quick experiment to test them, and integrating the results into your understanding of the problem, going back to the drawing board.
Your North Star: convince yourself a hypothesis is true or false. The key mindset is skepticism. Advice:
- Before testing a hypothesis, set a five-minute timer and brainstorm, "What are the ways this could be false?"
- Alternatively, write out the best possible case for your hypothesis and see where the argument feels weak.
- Try using an LLM with an anti-sycophancy prompt ("My friend wrote this and wants brutal feedback...") to red-team your arguments - it probably won’t work, but might be helpful
- Or set a 5 minute timer and brainstorm alternative explanations for your observations
You then want to convert these flaws and alternative hypotheses into concrete experiments. Experiment design is a deep skill. Honestly, I'm not sure how to teach it other than through experience. But one recommendation is to pay close attention to the experiments in papers you admire and analyze what made them so clever and effective. I also recommend that, every time you feel like you’ve (approximately) proven or falsified a hypothesis, adding them to a running doc of “things I believe to be true” with hypotheses, experiments, and results.
Using LLMs for Research Code
In my opinion, coding is one of the domains where LLMs are most obviously useful. It was very striking to me how much better my MATS scholars were six months ago than 12 months ago, and I think a good chunk of this is attributable by them having much better LLMs to use. If you are not using LLMs as a core part of your coding workflow, I think you're making a mistake.
- Use Cursor: It's VS Code with fantastic AI integration. Make sure to add the docs for libraries with @ so the AI has context. The $20/month plan is worth it, if possible, and there’s a free student version.
- Claude Code is tempting but bad for learning and iteration. I’d use it for throwaway things and first drafts - if the draft has a bunch of bugs, go read the code yourself/throw it away and start again. Cursor facilitates reading the AI’s code better than Claude code does IMO
- A caveat: If learning a new library (like in ARENA), first try writing things yourself. Use the LLM when stuck, not to replace the learning process.
- Later on, when thinking about writing up results, if key experiments were mostly vibe-coded, I recommend re-implementing them by hand to make sure no dumb LLM bugs slipped in.
Interlude: What’s New In Mechanistic Interpretability?
Feel free to skip to the “what should I do next” part
Things move fast in mechanistic interpretability. Newcomers to the field who've kept up from afar are often pretty out of date. Here's what I think you need to know, again, filtered through my own opinions and biases.
Avoiding Fads
This interlude is particularly important because the field often has fads: lines of research that are very popular for a year or so, make some progress and find many limitations, and then the field moves on. But if you’re new, and catching up on the literature, you might not realise. I often see people new to the field working on older things, that I don’t think are too productive to work on any more. Historical fads include:
Note that I am putting this after stage 2 because I think that for initial throwaway projects you should not be stressing about novelty and avoiding fads - your goal is just to learn. But as we move into stage 3 you should start to be a bit more mindful about choosing more exciting/impactful projects where possible.
Also, take these as nudges and recommendations, not as instructions. If there's a direction you believe in that fits the things I'm critiquing, maybe I'm just wrong, maybe your thing is an exception, go wild, see what happens.
What’s New In Mech Interp?
OK, so those are my hot takes on what not to do. What should you do? I think that some really cool new opportunities have opened up in mech interp over the last year, and newcomers may not have come across these. Here are some of the key themes in my favorite papers over the last year, that I’d love to see readers build on:
- Downstream tasks. If our North Star is to be able to do useful things with interpretability, a great way to test our techniques is on real-world tasks, to see if they beat baselines or enable us to do a new thing
- Even if we can't directly test on the tasks we most care about, if models can let us do new things in some domains, that's reasonable evidence that we're learning something real in a way that is hard to fake.
- One of my favourite examples is auditing games: Marks et al. is a fantastic paper that trained a model to have a hidden goal and then had different teams compete to find this goal, some with interpretability techniques, some without. Hypothesis: If interpretability does something real, then interp teams should win.
- This study itself was a bit ambiguous (the mech interp teams were 1st and 3rd, a black box team was 2nd), but I think this is a fantastic way to test interpretability techniques, and has a ton of potential for follow-on work evaluating interpretability in different kinds of games.
- The general idea is just that there is a model with some known hidden property and different techniques are used to see which can elicit it. Things like having different teams, allowed to use different techniques, etc. is just one implementation.
Model organisms: The auditing games paper was made possible by the fact that they were able to make a model with a hidden goal, a model organism to study. In general, we’re collecting techniques like synthetic document fine-tuning to make really interesting model organisms.
This kind of thing has a lot of potential! If we want to make a lie detector, a core challenge is that we don’t know how to test if it works or not. But if we can insert beliefs or deceptive behaviours into a model, many more projects become possible
A great intro project is playing around with open source model organisms, e.g. from Cywinski et al
- Practice on the real AGI Safety problems: Historically, interpretability could only practice on very dull toy problems like modular addition. But we now have models that exhibit complex behaviors that seem genuinely relevant to safety concerns, and we can just study them directly, making it far easier to make real progress.
- E.g. Rajamanoharan et al debunking assumed self-preservation, and Goldowsky-Dill et al probing for deception
- Weird behaviours: models can insider trade then lie about it, tell when they’re being evaluated (and act differently), fake alignment, reward hack, and more.
- Real-World Uses of Interpretability: Model interpretability-based techniques are starting to have genuine uses in frontier language models!
- Linear probes, one of the simplest possible techniques, are a highly competitive way to cheaply monitor systems for things like users trying to make bioweapons.
- I find it incredibly cool that interpretability can actually be useful, and kind of embarrassing that only a decade-old technique seems very helpful. Someone should do something about that. Maybe that someone could be you!
- This needs a very different kind of research: careful evaluation, comparison to strong baselines, and refinement of methods
- Attribution graph-based circuit analysis. The core problem with trying to analyze circuits in terms of things like a model's attention heads and layers is that often these things don't actually have a clear meaning. Attribution graphs use techniques like transcoders, popularized in Anthropic's model biology work, to approximate models with a computational graph with meaningful nodes.
See this cross-org blog post for the ongoing follow-on work across the community, and an open problems list I co-wrote!
- You can make and analyse your own attribution graphs on Neuronpedia
- Understanding model failures: Models often do weird things. If we were any good at interpretability, we should be able to understand these. Recently, we’ve seen signs of life!
- Meng et al on why some models think 9.8 < 9.11
A line of work studying emergent misalignment - why training models on narrowly evil tasks like writing insecure code turns them into Nazis - has found some insights. Wang et al found this was driven by sparse autoencoder latents associated with movie villains, and in Turner et al we found that the model could have learned the narrow solution, but this was in some sense less “efficient” and “stable”
Automated interpretability: Using LLMs to automate interpretability. We saw signs of life on this from Bills et al and Shaham et al, but LLMs are actually good now! It’s now possible to make basic interpretability agents that can do things like solve auditing games. And interpretability agents are the worst they’ll ever be.
Reasoning model interpretability: All current frontier models are reasoning models—models that are trained with reinforcement learning to think for a while before producing an answer. In my opinion, this requires a major rethinking of many existing interpretability approaches, and calls for exploring new paradigms. IMO this is currently being neglected by the field, but will become a big deal.
In Bogdan et al, we explored what a possible paradigm could look like. Notably, there are far more interesting and sophisticated black box techniques with reasoning models, like resampling the second half of the chain of thought, or every time the model says a specific kind of sentence, deleting and regenerating that sentence.
A Pragmatic Vision for Mech Interp
Attentive readers may notice that the list above focuses on work to do with understanding the more qualitative high-level properties of models, and not ambitious reverse engineering. This is largely because, in my opinion, the former has gone great, while we have not seen much progress towards the fundamental blockers on the latter.
I used to be very excited about ambitious reverse engineering, but I currently think that the dream of completely reverse engineering a model down to something human understandable seems basically doomed. My interpretation of the research so far is that models have some human understandable high-level structure that drives important actions, and a very long tail of increasingly niche and irrelevant heuristics and biases. For pragmatic purposes, these can be largely ignored, but not if we want things like guarantees, or to claim that we have understood most of a model. I think that trying to understand as much as we can is still a reasonable proxy for getting to the point of being pragmatically useful, but think it’s historically been too great a focus of the field, and many other approaches seem more promising if our ultimate goals are pragmatic.
In some ways, this has actually made me more optimistic about interpretability ultimately being useful for AGI safety! Ambitious reverse engineering would be awesome but was always a long shot. But I think we've seen some real results for pragmatic approaches to mechanistic interpretability, and feel fairly confident we are going to be able to do genuinely useful things that are hard to achieve with other methods.
Stage 3: Working Up To Full Research Projects
Once you have a few mini-projects done, you should start being more ambitious. You want to think about gaining the deeper (medium/slow) skills, and exploring ideation and distillation.
However, you should still expect projects to often fail, and want to lean into breadth over depth and avoid getting bogged down in an unsuccessful project you can’t bear to give up on. To resolve this tension, I recommend working in 1-2 week sprints. At the end of each sprint, reflect and make a deliberate decision: continue, or pivot? The default should be to pivot unless the project feels truly promising. It’s great to give up on things, if it means you spend your time even better! But if it’s going great, by all means continue.
This strategy should mean that you eventually end up working on something longer-term when you find something good, but don't just get bogged down in the first ambitious idea you tried.
I recommend reviewing the list of skills earlier and just for each one, reflecting for a bit on how on top of it you think you feel and how you could intentionally practice it in your next project. Then after each sprint, before deciding whether to pivot, take an hour or two to do a post-mortem: what did you learn, what progress did you make on different skills, and what would you do differently next time? Your goal is to learn, and you learn much better if you make time to actually process your accumulated data!
Key Research Mindsets
One way to decompose your learning is to think about research mindsets: the traits and mindsets a good researcher needs to have, that cut across many of these stages. See my blog post on the topic for more, but here's a brief view of how I'm currently thinking about it.
Skepticism/Truth-seeking: The default state of the world is that your research is false, because doing research is hard. Your north star should always be to find true insights
It generally doesn't come naturally to people to constantly aggressively think about all the ways their work could be false and make a good faith effort to test it. You can learn to do better than this, but it often takes practice.
This is crucial in understanding, somewhat important in exploration, and crucial in distillation.
A common mistake is to grasp at straws to find a “positive” result, thinking that nothing else is worth sharing.
In my opinion, negative or inconclusive results that are well-analyzed are much better than a poorly supported positive result. I’ll often think well of someone willing to release nuanced negative results, and poorly of someone who pretends their results are better than they are.
- Prioritization: Your time is scarce. Research involves making a bunch of decisions that are essentially searching through a high-dimensional space. The difference between a great and a mediocre researcher is being able to make these decisions well.
- If you have a good mentor, you can lean on them for this at first, but you will need to learn how to do this yourself eventually.
- This is absolutely crucial in exploration and ideation, but fairly important throughout.
- A good way to learn this one is to reflect on decisions you've made after the fact, eg in a sprint post-mortem, and think about how you could have made them better, and what generalisable lessons to take to the future
Productivity: The best researchers I've worked with get more than twice as much done as the merely good ones. Part of this is good research taste and making good prioritization decisions, but part of this is just being good at getting shit done.
Now, this doesn't necessarily mean pushing yourself until the point of burnout by working really long hours. Or cutting corners and being sloppy. This is about productivity integrated over the long term.
For example, sometimes the most productive thing to do is to hold off on starting work, set a 5 minute timer, brainstorm possible things to do next, and then pick the best idea
This takes many forms, and the highest priority for you:
Know when to write good code without bugs, to avoid wasting time debugging later, and when to write a hacky thing that just works.
Know the right keyboard shortcuts to move fast when coding.
Know when to ask for help and have people who can help you get unblocked where appropriate.
Be good at managing your time and tasks so that once you've decided what the highest priority thing to work on is, you in fact go and work on it.
Be able to make time to achieve deep focus on the key problems.
Exercise: Occasionally audit your time. Use a tool like Toggl for a day or two to log what you're doing, then reflect: where did time go? What was inefficient? How could I do this 10% faster next time?
The goal isn't to feel guilty, but to spot opportunities for improvement, like making a utility function for a tedious task.
- Knowing the literature: At this point, there’s a lot of accumulated wisdom (and a lot of BS) in prior papers, in mech interp and beyond.
- This cuts across all stages:
- In ideation, you don’t want to accidentally reinvent the wheel. And often great ideas are inspired by prior work
- In exploration, you want to be able to spot connections, borrow interesting techniques, etc
- In understanding, you want to know the right standards of proof to check for, the best techniques to use, alternative hypotheses (that may have been raised in other works), etc
In distillation, when writing a paper you’re expected to be able to contextualise it relative to existing work (i.e. write a related work section) which is important for other researchers knowing whether to care. And if you don’t know the standard methods of proof, key baselines everyone will ask about, key gotchas to check for etc, no one will believe your work.
- LLMs are an incredibly useful tool here. GPT-5 thinking or Claude 4 with web search are both pretty useful tools here, as are the slower but more comprehensive deep research tools (Note that Google's is available for free, as of the time of writing)
- I recommend using these regularly and creatively throughout a project.
- You don't necessarily need to go and read the works that get surfaced, but even just having LLM summaries can get you more awareness of what's out there, and over time you'll build this into deeper knowledge.
- Of course, when there does seem to be a very relevant paper to your work, you should go do a deep dive and read it properly, not just relying on LLM summaries.
- Don’t stress - deep knowledge of the literature takes time to build. But you want to ensure you’re on an upwards gradient here, rather than assuming the broader literature is useless
On the flip side, many papers are highly misleading/outright false, so please don’t just critically believe them!
Okay, so how does this all tie back to the stages of research? Now you're going to be thinking about all four. We'll start by talking about how to deepen your existing skills with exploration and understanding, and then we'll talk about what practicing ideation and actually writing up your work should look like.
Deepening Your Skills
You’ll still be exploring and understanding, but with a greater focus on rigor and the slower skills. In addition to the thoughts when discussing mindsets above, here’s some more specific advice
- Deeper Exploration is about internalizing the mindset of maximising productivity, which here means maximising information gain per unit time. Always ask, "Am I learning something?"
- Avoid Rabbit Holes: A common mistake is finding one random anomaly and zooming in on it. Knowing when to pivot is crucial. Set a timer every hour or two to zoom out and ask if you’re making progress.
- I recommend any time you notice yourself feeling a bit stuck or distracted or off track, setting a five minute timer and thinking about what could I be doing next, what should I be doing next, and am I doing the most important thing?
- Avoid Spreading Yourself Too Thin: Doing lots of things superficially means none of them will be interesting.
- If you have spent more than five hours without learning something new, you should probably try a different approach
- And if you have spent more than two days without learning something new, you should seriously consider pivoting and doing something else.
- To practice prioritization, be intentional about your decisions: write down why you think an experiment is the right call, and later reflect on whether you were right. This makes your intuitions explicit and easier to update.
- Deeper Understanding is about practicing skepticism and building a bulletproof case. Red-team your results relentlessly.
- Some experiments are much more impactful and informative than others! Don't just do the first experiment that pops into your head. Think about the key ways the hypothesis could be false, and how you could test that. Or about whether a skeptic could explain away a positive experimental results
- A useful exercise is imagining you're talking to a really obnoxious skeptic who keeps complaining that they don't believe you and coming up with arguments for why your thing is wrong. What could you do such that they don't have a leg to stand on?
- Of course, there's also an element of prioritization. Sometimes a shallow case that could be wrong is the right thing to aim for, if you’re working on an unimportant side claim/something that seems super plausible on priors, at which point you should just move on and do something else more interesting.
- Exercise: To practice spotting subtle illusions, try red-teaming papers you read, thinking about potential flaws, and ideally run the experiments yourself.
Doing Good Science
Practicing Ideation
Okay, so you want to actually come up with good research ideas to work on. What does this look like? I recommend breaking this down into generating ideas and then evaluating them to find the best ones.
To generate ideas, I'd often start with just taking a blank doc, blocking out at least an hour, and then just writing down as many ideas as you can come up with. Aim for quantity over quality. Go for at least 20.
There are other things you can do to help with generation:
- Throughout your previous sprints, every time you had an idle curiosity or noticed something weird, write it down in one massive long-running doc.
- Likewise, when reading papers, note down confusions, curiosities, obviousnesses to do.
Okay, so now you have a big list. What does finding the best ones look like?
- Ideally, if you have a mentor or at least collaborators, you can just ask them to rate them.
- If you do this, rate them yourself privately out of 10 before you look at their responses. Compare them and every time you have substantially different numbers, talk to the mentor and try to figure out why your intuitions disagree. This is a great source of supervised data for research taste.
- Even if you don’t have a mentor, I think that just going through, rating each idea yourself based on gut feel and sorting is as good a way to prune down a long list as any
- For the top few, I recommend trying to answer a few questions about them.
- What would success look like here?
- How surprised would I be if I did this for a month and nothing interesting had happened?
- What skills does this require? Do I have them/could I easily gain them?
- What models, data, computational resources, etc. does this require?
- How does this compare to what the most relevant prior work did? Can I check for prior work and see if anything relevant comes up?
Research Taste Exercises
Gaining research taste is slow because the feedback loops are long. You can accelerate it with exercises that give you faster, proxy feedback. (Credit to Chris Olah for inspiration here)
- If you have a mentor, query their taste for fast data and try to imitate it. Concretely:
- Learning from papers as "offline data": When you read a paper, don't just passively consume it. Read the introduction, then stop. Try to predict what methods they used and what their key results will be. Then, continue reading and see how your predictions compare. Analyze why the authors made different choices. This trains your intuition on a much larger and faster dataset than your own research.
It’s also worth dwelling on what research taste actually is. See my post for more, but I break it down as follows:
- Intuition (System 1): This is the fast, gut-level feeling - what people normally think of when they say research taste. A sense of curiosity, excitement, boredom, or skepticism about a direction, experiment, or result.
- Conceptual Framework (System 2): This is deep domain knowledge and understanding of underlying principles.
- Strategic Big Picture: Understanding the broader context of the field. What problems are important? What are the major open questions? What approaches have been tried? What constitutes a novel contribution?
Write up your work!
At this stage, you should be thinking seriously about how to write up your work. Often, writing up work is the first time you really understand what a project has been about, or you identify key limitations, or experiments you forgot to do. You should check out my blog post on writing ML papers for much more detailed thoughts (which also apply to high-effort blog posts!) but I'll try to summarize them below.
Why aim for public output?
If producing something public is intimidating, for now, you can start by just writing up a private Google Doc and maybe share it with some friends or collaborators. But I heavily encourage people to aim for public output where they can. Generally, your research will not matter if no one reads it. The goal of research is to contribute to the sum of human knowledge. And if no one understands what you did, then it doesn't matter.
Further, if you want to pursue a career in the space, whether a job, a PhD, or just informally working with mentors, public research output is your best credential. It's very clear and concrete proof that you are competent, can execute on research and do interesting things, and this is exactly the kind of evidence people care about seeing if they're trying to figure out whether they should work with you, pay attention to what you're saying, etc. It doesn’t matter if you wrote it in a prestigious PhD program or as a random independent researcher, if it’s good enough then people care.
There are a few options for what this can look like:
- A blog post (e.g. on a personal blog or LessWrong) - the simplest and least formal kind
An Arxiv paper - much more legible than a blog post, and honestly not much extra effort if you have a high-quality blog post
A workshop paper (i.e. something you submit for peer review to a workshop, typically part of a major ML conference, the bar is much lower than for a conference paper)
A conference paper (the equivalent of top journals in ML, there’s a reasonably high quality bar, but also a lot of noise)
If this all seems overwhelming, starting out with blog posts is fine, but I think people generally overestimate the bar for arxiv or workshop papers - if you think you learned something cool in a project, this is totally worth turning into a paper!
How to write stuff up?
The core of a paper is the narrative. Readers will not take away more than a few sentences worth of content. Your job is to make sure these are the right handful of sentences and make sure the reader is convinced of them.
You want to distill your paper down into one to three key claims (your contribution), the evidence you provide that the contribution is true, the motivation for why a reader should care about them, and work all of this into a coherent narrative.
Iterate: I'm a big fan of writing things iteratively. You first figure out the contribution and narrative. You then write a condensed summary, the abstract (in a blog post, this should be a TL;DR/executive summary - also very important!). You then write a bullet point outline of the paper: what points you want to cover, what evidence you want to provide, how you intend to build up to that evidence, how you want to structure and order things, etc. If you have mentors or collaborators, the bullet point outline is often the best time to get feedback. Or the narrative formation stage, if you have an engaged mentor. Then write the introduction, and make sure you’re happy with that. Then (or even before the intro) make the figures - figures are incredibly important! Then flesh it out into prose. People spend a lot more time reading the abstract and the intro than the main body, especially when you account for all the people who read the abstract and then stop. So you should spend a lot more time per unit word on those.
LLMs: I think LLMs are a really helpful writing tool. They're super useful for getting feedback, especially if writing in an unfamiliar style like an academic ML paper may be for you. Remember to use anti-sycophanty prompts so you get real feedback. However, it's often quite easy to tell when you're reading LLM written slop. So use them as a tool, but don't just have them write the damn thing for you. But if you e.g. have writer’s block, having an LLM help you brainstorm or produce a first draft for inspiration, and can be very helpful.
Common mistakes
- The reader does not have context: Your paper will be clear in your head, because you have just spent weeks to months steeped in this research project. The reader has not. You will overestimate how clear things are to the reader, and so you should be massively erring in the other direction and spelling everything out as blatantly as possible.
- This is an incredibly common mistake - assume it will happen to you
- The main solution is to get feedback from people with enough research context that they can actually engage and who are also willing to give you substantial negative feedback.
- Notice the feeling of surprise when people are confused by something you thought was clear. Try to understand why they were confused and iterate on fixing it until it's clear.
- Writing is not an afterthought: People often do not prioritize writing. They treat it like an annoying afterthought and do all the fun bits like running experiments, and leave it to the last minute.
- Acknowledge limitations: There is a common mistake of trying to make your work sound maximally exciting. Generally, the people whose opinions you most care about are competent researchers who can see through this kind of thing
- Good writing is simple: There's a tendency towards verbosity or trying to make things sound more complex and fancy than they actually are, so they feel impressive. I think this is a highly ineffective strategy
- Remember to motivate things: It will typically not be obvious to the reader why your paper matters or is interesting. They do not have the context you do. It is your job to convince them, ideally in the abstract or perhaps intro, why they should care about your work, lest they just give up and stop reading.
Mentorship, Collaboration and Sharing Your Work
A common theme in the above is that it's incredibly useful to have a mentor, or at least collaborators. Here I'll try to unpack that and give advice about how to go about finding one.
Though it's also worth saying that many mentors are not actually great researchers and may have bad research taste or research taste that's not very well suited to mech interp. What you do about this is kind of up to you.
So what does a research mentor actually do?
A good mentor is an incredible accelerator. Dysfunctional as academia is, there is a reason it works under the apprenticeship-like system of PhD students and supervisors. When I started supervising, I was very surprised at how much of a difference a weekly check in could make! Here’s my best attempt to breakdown how a good mentor can add value:
Advice on finding a mentor
Here are some suggested ways to get some mentorship while transitioning into the field. I discuss higher commitment ways, like doing a PhD or getting a research job, below.
Note: whatever you do to find a mentor, having evidence that you can do research yourself, that is, public output that demonstrates ability to self-motivate and put in effort, and ideally demonstrates actually interesting research findings, is incredibly helpful and should be a priority.
Mentoring programs
I think mentoring programs like MATS are an incredibly useful way into the field, you typically do a full-time, several month program where you write a paper, with weekly check-ins with a more experienced researcher. Your experience will vary wildly depending on mentor quality, but at least for my MATS scholars, often people totally new to mech interp can publish a top conference paper in a few months. See my MATS application doc for a bunch more details.
There’s a wide range of backgrounds among people who do them and get value - people totally new to a field, people with 1+ years of interpretability research experience who want to work with a more experienced mentor, young undergrads, mid-career professionals (including a handful of professors), and more.
MATS 9.0 applications are open, due Oct 2 2025, and mine close on Sept 12.
Other programs (which I think are generally lower quality than MATS, but often still worth applying to depending on the mentor)
Sending A Good Cold Email
You can also take matters into your own hands and try to convince someone to be your mentor. Reaching out to people, ideally via a warm introduction, but even just via a cold email, can be highly effective. However, I get lots of cold emails and I think many are not very effective, so here's some advice:
- Don't just email the most prominent people. A lot of people will just email the most prominent people in the field and ask for mentorship. This is a bad plan! These people are very busy and they also get lots of emails. I just reflexively respond to any email requesting mentorship with “please apply to my MATS cohort”.
- However, there are lots of less prominent people who can provide a bunch of useful mentorship. These people are much more likely to be excited to get a cold email, to have time to engage, potentially even the spare capacity to properly mentor a project.
- I think that many people who've recently joined my team or people who worked on a great paper with me during MATS are able to add a lot of value to people new to the field. And I would recommend reaching out to them!
- For example, Josh Engels, a new starter on my team, said he would happily receive more cold emails (as of early Sept 2025).
- As a general heuristic, email first authors of papers, not fancy last authors.
- Start small: Don't email someone you've never interacted with before asking if they want to kind of officially mentor you on some project. That's a big commitment.
- It's much better to be like, I'd be interested in having a chat about your paper or my work building on your paper.
- Or just asking if they're down to have a chat giving you feedback on some project ideas, etc.
- And if this goes well, it may organically turn into a more long-term mentoring relationship!
- Proof of work: Demonstrate that you are actually interested in this person specifically, not just spamming tons of people.
- Show that you've engaged with their work, say something intelligent about it, have some questions.
- In the era of LLMs, this is less of a costly signal that you've actually taken an interest in this person specifically than it used to be, admittedly
- But linking to some research you did building on their work I think is still reasonably costly, and very flattering to people.
- Prioritize aggressively. Assume the reader will stop reading at any moment, so put your most critical and impressive information first.
- Explain who you are: If you're emailing someone who gets more emails than they have capacity to respond to, they're going to be prioritizing. A key input into this is just who you are, what have you done, have you done something interesting that shows promise, do you have relevant credentials, etc. I personally find it very helpful if people just say the most impressive things about them in the first sentence or two.
- To do this without seeming arrogant, you could try: "I'm sure you must get many of these emails. So to help you prioritise, here's some key info about me"
- Use bolding for key phrases to make your email easily skimmable.
- Be concise. One thing I would often appreciate is a short blurb summarizing your request with a link to a longer document for details if I'm interested.
- Quick requests: Generally, my flow when reading emails is that I will either immediately respond or never look at it again. I'm a lot more likely to immediately respond if I can do so quickly. If you do want to email a busy person, have a clear, concrete question up front that they might be able to help with.
Much easier than finding a mentor is finding collaborators, other people to work on the same project with, or just other people also trying to learn more about mech interp, who you can chat with and give each other feedback:
- In-Person: Local AI Safety hubs (London, Bay Area, etc.), University groups, ML conferences (e.g., the NeurIPS Mech Interp workshop I co-organize), EAG/EAGx conferences.
- If you’re a student, see if there’s a lab at your university that has some people interested in interpretability. There may be interested PhD students even if no professor works on it
- Online: These are also good places to meet people! I recommend sharing work for feedback, or just asking about who’s interested in what you’re interested in, and trying to DM the people who engage/seem interested, and seeing what happens
Staying up to date: Another common question is how to stay up to date with the field. Honestly, I think that people new to the field should not worry that much about this. Most new papers are irrelevant, including the ones that there is hype around. But it's good to stay a little bit in the loop. Note that the community has substantial parts both in academia and outside, which are often best kept up with in different ways.
- LessWrong and the AlignmentForum are a reasonable place to keep up to date with the less academic half
- Twitter is a confusing, chaotic place that is an okay place to keep up with both. It's a bit unclear who the right people to follow.
- Chris Olah doesn't tweet much, but it's high quality when he does.
- I will tweet about all of my interpretability work and sometimes others.
Careers
Where to apply
Applying for grants
For people trying to get into mech interp via the safety community, there are some funders around open to giving career transition grants to people trying to upskill in a new field like mech interp. Probably the best place I know of is Open Philanthropy's Early Career Funding.
Explore Other AI Safety Areas
Mech interp isn't the only game in town! There’s other important areas of safety like Evals, AI Control, and Scalable Oversight, the latter two in particular seem neglected compared to mech interp. The GDM AGI Safety Approach gives an overview of different parts of the field. If you’re doing this for safety reasons, I’d check if there’s other, more neglected subfields, that also appeal to you!
What do hiring managers look for
Leaving aside things that apply to basically all roles, like whether this person has a good personality fit (which often just means looking out for red flags), here’s my sense of what hiring managers in interpretability are often looking for.
A useful mental model is that from a hiring manager's perspective, they're making an uncertain bet with little information in a somewhat adversarial environment. Each applicant wants to present themselves as the perfect fit. This means managers need to rely on signals that are hard to fake. But it’s quite difficult to get that much info on a person before you actually go and work with them a bunch.
Your goal as a candidate is to provide compelling, hard-to-fake evidence of your skills. The best way to do that is to simply do good research and share it publicly. If your research track record is good enough, interviews may just act as a check for red flags and to verify that you can actually write code and run experiments well.
Key skills:
- Research Skills: A track record of completing end-to-end projects is the best signal. Papers are a great way to show this.
- Research taste: The ability to come up with great research ideas and drive them to completion is rare and very valuable.
- Experiment design: Can they design good experiments and make their research ideas concrete and convert them into actions?
- Conceptual Understanding of Mech Interp: Do you get the key ideas and know the literature?
Productivity and Conscientiousness: This is a very hard one to interview for, but incredibly important. A public track record of doing interesting things is a good signal, as are strong references from trusted sources.
- Engineering Skills: Can you work fluently in a Python notebook? Can you write experiment code fast and well? Can you get things done? Do you understand the standard gotchas?
- Deep engineering skill: Beyond hacking together experiments, can you navigate large, complex codebases, write maintainable code, design complex software projects, etc?
- This is much more important if doing research inside a larger lab or tech company than as an independent researcher or academic.
- One of the most common reasons we don't hire seemingly promising researchers onto my team is because they lack sufficiently strong engineering skills.
- Obviously, LLMs are substantially changing the game when it comes to engineering skills, but I think deep engineering skills will be much harder to automate than shallow ones, unfortunately.
- Unfortunately, I don’t have great advice on how to gain these other than working in larger and more complex codebases and learning how to cope. Pair programming with more experienced programmers can be a great way to transfer tacit knowledge
- Skepticism: Can you constructively engage with research and critically evaluate it? In particular, can you do this to your own research? Good researchers need to be able to do work that is true.
Should you do a PhD?
I don't have a PhD (and think I would have had a far less successful career if I had tried to get one) so I'm somewhat biased. But it's a common question. Here are the strongest arguments I’ve heard in favour:
- You get extremely high autonomy. If you want to spend years going deep on a niche topic that no industry lab would fund, a PhD is one of the only ways to do it.
- It's a great environment to cultivate the ability to set your own research agenda. This is a crucial and difficult skill that is harder to learn in industry, where agendas are often set from the top down (though this varies a lot between team).
And here are the reasons I think it's often a bad idea:
- The opportunity cost is immense. You could spend 4-6 years gaining direct, relevant experience in an industry lab.
- Academic incentives can be misaligned with doing impactful research, e.g. pressure to publish meaning you’re discouraged from admitting to the limitations of your work.
- The quality of supervision varies wildly, and a bad supervisor can make your life miserable.
- Quality of life: The pay is generally terrible, which may or may not matter to you, and you may only get places in a different city/country than you’d prefer.
But with all those caveats in mind, it’s definitely the right option for some! My overall take:
- The key thing that matters is mentorship, being in an environment where you are working with a better researcher, and learning from them.
- PhDs are often a good way of getting this. But if you can gain this by another way, plausibly you should go to that instead. PhDs have a lot of downsides too.
- Generally, the variance between supervisors and between managers in industry will dominate the academia versus industry differences, and thus you should pay a lot of attention to who exactly would be managing you.
- For a PhD, try to speak to your potential supervisor’s students in a private setting. If they say pretty bad things, that's a good reason not to go for the supervisor.
- A common mistake is optimising for the most prestigious and famous supervisor when you often want to go for the ones who will have the most time for you, which anti-correlates.
- A common mistake is people feeling they need to finish PhDs. But if you sincerely believe that the point of a PhD is to be a learning environment, then why would the formal end of the PhD be the optimal time to leave? It's all kind of arbitrary.
- IMO, at least every six months, you should seriously evaluate what other opportunities you have, try applying for some things and be emotionally willing leave if a better opportunity comes along (taking into account switching costs).
- Note that often you can just take a year's leave of absence and resume at will.
Relevant Academic Labs
I’m a big fan of the work coming out of these two, they seem like great places to work:
- David Bau (Northeastern)
- Martin Wattenberg & Fernanda Viegas (Harvard)
Other labs that seem like good places to do interpretability research (note that this is not trying to be a comprehensive list!):
- Yonatan Belinkov (Technion)
- Jacob Andreas (MIT)
- Jacob Steinhardt (Berkeley)
- Ellie Pavlick (Brown)
- Victor Veitch (UChicago)
- Robert West (EPFL)
- Roger Grosse (Toronto)
- Mor Geva (Tel Aviv)
- Sarah Wiegreffe (Maryland)
- Aaron Mueller (Boston University)
Thanks a lot to Arthur Conmy, Paul Bogdan, Bilal Chughtai, Julian Minder, Callum McDougall, Josh Engels, Clement Dumas, Bart Bussmann for valuable feedback
1