Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples.
I spent the better part of 2026 in empirical AI safety research apprenticeships. I was mentored by Alex Turner and Alex Cloud as part of MATS 9 Team Shard, where my collaborator and I worked on finetuning-based misalignment auditing. I then interned at Redwood Research, managed by Aryan Bhatt, where I worked on sabotage capability evaluations.
These thoughts are indebted to my mentors, collaborators, friends, and colleagues, but they are not direct reflections of their views.
Thank you to Alex Cloud, Alex Turner, Aryan Bhatt, Jeremy Huff, and especially to Keshav Shenoy for feedback on the piece.
Better models of the problem speed up empirical iteration
Fast iteration is one of the key virtues of an empirical safety researcher. At first, I didn't find it obvious what that meant or how I should get better at this skill. I imagined a sprinter, doing one looped motion at a far quicker pace than everyone else. I now think that’s the wrong mental picture, and going fast is more like being an efficient grappler–agile, strategic, calibrated about where to put pressure, and fluent enough in the core techniques that the right moves come instinctively.
A large part of iterating fast is being able to pick out the most uncertainty-reducing, high-information-value questions in a problem, and then generate a bunch of good ideas to answer these questions quickly. Part of this arises from being generically good at thinking, but a lot of this skill is about using the right frames. That comes with intentional practice.
One useful frame for intentionally improving this skill is the scientific process: developing a predictive, falsifiable model, designing experiments to test it, and updating the model from data. A lot of empirical research might seem focused on getting things to work (e.g. showing a phenomenon happens, or showing a method is effective), which suggests an engineering, rather than science-centred approach. In my experience however, people with better models will pivot faster, do experiments more efficiently, and come up with better ideas.
Develop predictive models and update them
A predictive model can be wrong but still useful. It supplies a starting belief to update from, and helps generate ideas. Ptolemy’s geocentric model worked as well as the heliocentric model given the quality of observation data he had access to.
The persona selection model (PSM) is a good example of a model that doesn’t try to be mathematically rigorous but is still generative and partially falsifiable. PSM says, in brief, that in pre-training AIs learn to simulate many personas, and in post-training the assistant persona is selected for and further refined. It suggests taking an anthropomorphic lens. PSM inspired efforts in shaping pre-training priors such as Alignment Pretraining. New evidence can update PSM: agent behaviours in 2026 suggest that different training pressures create split personas that activate in different contexts. For further discussions, see thoughts on the persona selection model.
To give an example from my work which is closer to classic ML, I was recently working on testing the theory that sabotage capability has a simple latent structure that makes model performance on control evals predictable from a few parameters. Our model helped generate function classes to test, and favoured exploration into simpler function families. We found that a simple function with few parameters indeed explained our held-out data, which was evidence in support of our model. Had it been the case that thorough, principled search over simple function classes failed to predict held-out data, I would have updated against the model.
Update with uncertainty in mind
When updating predictive models based on data, error bars usually do not account for all relevant sources of uncertainty, and it’s good to keep in mind the full picture when updating.
To illustrate, say I want to learn how well a weak model detected a stronger model’s stealthy attacks in an AI control context. It turns out that the weak monitor caught 80% of attacks at a small auditing budget in LinuxArena. My prior had a mean of 50%. If I just look at the 80% number, the monitor model punches surprisingly above its weight.
If I only had five attack samples sampled from the same main-side task pair, my error bars are huge, and I probably shouldn’t update from 50% immediately to 80%. I might update to 55%.
If I ran 100 samples and the number is still 80%, I would be less uncertain, but since I care about how well the weak model did in different situations, I would want to run this experiment with a variety of main-side task pairs.
If I do that and then still get 80%, then I start being more confident that my weak model really was that good. Still, I might worry that LinuxArena does not capture the distribution I actually care about. That’s harder to estimate, and I may not be able to quantify my uncertainty, so I should mentally register that I now have some uncertainty about the world that my error bars don’t account for.
At this moment, AIs are not very good at being principled about accounting for sources of uncertainty. I expect that they will become better very soon, but in the meantime, it’s worth checking whether the AI gave a reasonably complete and fair picture of the uncertainty.
Predict senior researchers’ research takes
One of the most useful exercises for improving research taste is intentionally trying to predict what senior researchers would say when discussing research.
A few ways to practice:
If my mentor is happy with recording the meeting, I would sometimes ask Claude to quiz me on key points my mentor made.
I’ve heard of people making Anki cards from mentor meetings.
Some researchers (like Neel Nanda here) post videos of themselves doing research, and it can be a nice exercise to pause the video and predict what they'll do or say next.
While trying to predict senior researchers, I often notice intellectual idiosyncrasies. It’s helpful to understand why these habits of mind are effective while being clear about their limitations. For example, Alex Cloud likes to do the MNIST-test before picking experiments:
Any time you come across a clever-sounding machine learning idea, ask: "can I apply this to MNIST?" If not, then it's not a good idea. If so, run an experiment to see if it works. If it doesn't, then it's not a good idea. If it does, then it might be a good idea, and you can continue as usual to more realistic experiments or theory. Source: Omniscaling to MNIST.
The MNIST-test is useful because it selects for methods that work on neural networks of varying scales, not just current-day AIs. However, as Cloud notes in the post, the selection heuristic can miss many good ideas that depend on capabilities or structure absent from the MNIST setting.
Use metacognitive strategies to make predicting and updating habitual
One piece of advice I received was ‘predict the outcome of your experiment’. I think this is good advice. Pre-registering experimental predictions by using e.g. Fatebook helps me notice when I should be surprised and update my model of the world.
Remembering to predict and update requires deliberate effort initially. Ideally, we want this act to be intuitive, habitual, and agile. I find rationalist metacognitive strategies to be very helpful for building mind habits. I especially recommend people carefully read Tuning your Cognitive Strategies.
Metacognitive techniques look like doing structured introspection: noticing thought and the felt sense of thought, making increasingly granular observations about cognitive processes, and then deliberately engineering them. In general, noticing a thought pattern is the main bottleneck to refining it.
To give an example, say I am trying to predict the test set in a train/test split, and I also have available another train set that’s bigger but more off-distribution. I find that if I use this other train set, I do better on the target test set. I might notice a mild sense of surprise. At this point, I should encourage my mind to ask, why am I surprised? I want to bring my implicit model into the front stage, and think about how my model would change if this were true. Then I might want to be a bit skeptical, and run through possibilities for things I or my agent might’ve done wrong. Maybe I go double-check these. These thoughts are not heavy-duty. They should be System 1. The emphasis here is that noticing surprise should naturally lead to a whole sequence of thoughts and reactions.
Spend more effort on research communication
Communication quality bottlenecks the reception, adoption, and thus expected impact of the research. Researchers in and out of the community have emphasized this point:
Richard Hamming: I believed, in my early days, that you should spend at least as much time in the polish and presentation as you did in the original research. Now at least 50% of the time must go for the presentation. It's a big, big number. Source: You and Your Research.
There are great resources out there for writingpapers and giving talks. I will focus on research updates, which are more informal than papers, talks, posters, and blog posts. Examples of research updates include weekly research slides, Slack updates, and short progress talks.
Writing good research updates helps mentors allocate their limited attention to discussions and decisions that actually matter, instead of spending most of their time getting up to speed with the project. See Fig 1 for an illustrative model.[1]
Fig 1: The mentor starts off with a bit more context than the IC (individual contributor), but the IC continuously gains more context, so it’s particularly important for the IC to get the mentor back in the game quickly and regularly with every research update. I drew a sketch and Astra made it nice.
My sense is the majority of people starting empirical safety research aren’t calibrated about the importance of good research updates and should front-load effort into improving this skill. This was true in my case and seems true after talking to my peers.
Readers of research updates are mentors, managers, collaborators, and colleagues. They are busy, so they are not caught up on the low-level details, but know a lot about research, AI safety, and the general direction. So, good research updates should be both digestible and rigorous.
To illustrate what I mean, I’ll say more about what makes good Slack update threads and plots. For advice on making good slides, see Tips On Empirical Research Slides.
Writing good Slack update threads
Good update threads vary in style, but usually they do the following really well:
Heavy goal-posting and keywords emboldening. A good update doesn’t have to be structured with bolded subheadings, but usually it’s obvious from the first 5 seconds of reading what the keyword and key sections are.
A starting section that functions as a TL;DR. It answers the question of (1) why the reader should care, which, for results updates, could be as simple as ‘good/mixed/negative results on x problem that we’ve established is important for the project’ and (2) what the gist of the update is (e.g. ‘x intervention probably doesn’t work on y model in z setting’).
Calibrated about the knowledge level of the audience. Since the reader is high-context, there’s no need to repeatedly define or simplify known vocabulary for the project (e.g. for my MATS projects, my readers know what I mean by elicitability). But the reader won’t know the specific details of an experiment that’s being run, and background would be useful.
Includes all the necessary details in the update so the mentor can flag any mistakes. The mentor may not read everything, but the update should be prepared to answer questions that come up. This means reporting data and train/test splits, method of calculating error bars, key hyperparameters, and all non-obvious decisions made in the experiment.
Completely human written. At this moment, I don’t think AI should be used to write Slack update threads, since AIs are way less good than humans at being calibrated about how digestible an update is. For example, Claude loves to invent new words to describe objects in the project - I think these words are often hard to understand and I avoid using them when communicating to other humans.
At first, one way of reaching the rigorous and skimmable sweet spot for writing Slack threads is to write a rigorous research update, then spending some time making the update digestible, then combining them. By 'at first' I mean the first <5 times. I think people should expect to get way better at this very quickly. At the start of MATS, I found the idea of spending an entire day polishing my research slides excessive, but in hindsight front-loading communication level-ups would clearly have been worth it.
Making good plots
A plot is an argument in miniature and it's what people read. When I skim a paper, I usually read the title, the abstract, and then the main figures.
Most early-career empirical safety researchers should spend more time making the plots nice. Whenever I feel like I’ve put in enough effort on making my plots nice, it’s usually the case that I should’ve spent even more time.
There’s somegreatadvice on plotting which I will not reiterate. What I do want to talk about is how I think about what makes plots easy to understand.
I’ve heard the advice that plots should be simple. It’s worth emphasizing that simple means simple relative to the existing scientific understanding. To my 10 year old self a calibration plot would have been not simple, but to the scientific community it’s well known, recognizable, and hence simple.
Plots should fit into templates. Often, bar plots say “this one is higher than this one”, diverging line plots say “this line goes up, this one doesn't”, and Pareto frontier plots usually say “we push the Pareto curve”. I give more examples in Figure 2.
Fig 2: Plots that fit existing templates are way easier to understand. I give some examples of plot templates and what they tend to communicate. The data is synthetic and meant to illustrate the main points. I made the figures with Astra.
Be calibrated about the process
Researchers frequently make judgments about what progress looks like, how much their time is worth, or whether some investment in energy or resources is worth it. I find it helpful to model these beliefs about these strategic considerations explicitly so we can form more accurate beliefs about them. I give some examples below:
Good confusion feels different from bad confusion
I have felt both good confusion and bad confusion. Good confusion reflects real uncertainty in the world; bad confusion is where I don't know what's going on in the project.
An example of good confusion: I want to find a good metric, but I don’t know what the right one is. The quantity I’m trying to estimate is conceptually fuzzy, and I don’t even know if it’s a natural quantity in the world. I need to try a bunch of stuff to slowly clarify if this quantity is well-defined, existent, and natural.
An example of bad confusion: I don’t know if I’m in the right stage to wrap up the project, or if I should continue exploring more directions within the paper, and I’m doing both at the same time while not knowing what I should expect to have done in two weeks.
I find that the felt sense is pretty obviously different for both, so it’s useful to develop an intuitive handle on what state I’m in, and then practice being comfortable with good confusion while reducing bad confusion.
Sometimes it’s better to burn money than time
I find it helpful to estimate the expected value of a particular task in the next few hours / days / weeks, and then ask if the appropriate amount of money and time is being used. For example, at both MATS and Redwood, there were times where I erred on the side of using too little money while spending way too much time on things that don't matter. In those cases, I probably should have traded time for money by, for example, using Modal instead of a cheaper GPU provider because it fanned out faster, or simply running more samples of things because higher sample size was higher information value in that scenario.
Small time commitments become costly in attention
The impact of empirical projects is heavy-tailed–a very good project is way more useful than a good project. Hence, protecting focus matters a lot.
When focusing on one project, I find that small work-related time commitments such as an extra 1-2 hours of meeting a week become way more costly in attention. If I take 3 meetings a day, then taking 4 meetings does not change the type of work I do. If I normally take no meetings, then taking a meeting will fragment my day, which disrupts focus. This Paul Graham post makes the full point: Maker's Schedule, Manager's Schedule.
Know which level of ‘being done’ to aim for
Projects with different levels of completion requirements can demand an order of magnitude more or less effort. I think researchers should be clear about the level of ‘done’ they want to achieve and plan accordingly.[2]
Here's a frame I find useful:
An epistemic-complete empirical project can defend its central claim, but is messy at the edges. It’s written for co-workers: people can talk about the results over lunch at work and trust that it’s correct enough to build on.
A paper-complete project is way more fleshed out, does a bunch of ablations and robustness checks, and defends its claims carefully. It’s ready to be defended amongst skeptics.
My sense is that a paper-complete project is 1/2-1 OOM higher effort compared to an epistemic-complete project. Take Alignment Faking as an example. Ryan Greenblatt demonstrated the core results, and it took a much larger team, mostly of Anthropic researchers, for the project to reach a paper-complete state.
There are merits to choosing different levels of ‘done’. Epistemic-complete projects finish way faster, while good papers reach people beyond the safety community, and are good for careers or visa applications. People who want to be done after epistemic-completeness should resist the urge to explore more directions after they are convinced of the core results; people who want to write good papers should expect the process to take months, even after the core results have been established.
Closing Thoughts
To practice these skills, I highly recommend assigning a generous fraction of time to writing things down.[3] I kept an informal running list of research lessons in Obsidian, which helped me consolidate what I learned. I wrote Google Docs about my confusions, where I think the project is going, and where my current model stands. I wish I’d written even more.
Most of the lessons I shared are downstream of treating technical AI safety research as a science: developing a model of the world, making predictions, empirically testing these predictions, and then updating. Having a wrong model is better than having an unfalsifiable one. In the words of Francis Bacon: “Truth emerges more readily from error than from confusion.”
This model is illustrative, and adopts a narrow take on ‘context’: at all points, mentors will in all likelihood have good object-level takes that the IC doesn’t due to having better research taste, but this sketch doesn’t account for that.
Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples.
I spent the better part of 2026 in empirical AI safety research apprenticeships. I was mentored by Alex Turner and Alex Cloud as part of MATS 9 Team Shard, where my collaborator and I worked on finetuning-based misalignment auditing. I then interned at Redwood Research, managed by Aryan Bhatt, where I worked on sabotage capability evaluations.
These thoughts are indebted to my mentors, collaborators, friends, and colleagues, but they are not direct reflections of their views.
Thank you to Alex Cloud, Alex Turner, Aryan Bhatt, Jeremy Huff, and especially to Keshav Shenoy for feedback on the piece.
Better models of the problem speed up empirical iteration
Fast iteration is one of the key virtues of an empirical safety researcher. At first, I didn't find it obvious what that meant or how I should get better at this skill. I imagined a sprinter, doing one looped motion at a far quicker pace than everyone else. I now think that’s the wrong mental picture, and going fast is more like being an efficient grappler–agile, strategic, calibrated about where to put pressure, and fluent enough in the core techniques that the right moves come instinctively.
A large part of iterating fast is being able to pick out the most uncertainty-reducing, high-information-value questions in a problem, and then generate a bunch of good ideas to answer these questions quickly. Part of this arises from being generically good at thinking, but a lot of this skill is about using the right frames. That comes with intentional practice.
One useful frame for intentionally improving this skill is the scientific process: developing a predictive, falsifiable model, designing experiments to test it, and updating the model from data. A lot of empirical research might seem focused on getting things to work (e.g. showing a phenomenon happens, or showing a method is effective), which suggests an engineering, rather than science-centred approach. In my experience however, people with better models will pivot faster, do experiments more efficiently, and come up with better ideas.
Develop predictive models and update them
A predictive model can be wrong but still useful. It supplies a starting belief to update from, and helps generate ideas. Ptolemy’s geocentric model worked as well as the heliocentric model given the quality of observation data he had access to.
The persona selection model (PSM) is a good example of a model that doesn’t try to be mathematically rigorous but is still generative and partially falsifiable. PSM says, in brief, that in pre-training AIs learn to simulate many personas, and in post-training the assistant persona is selected for and further refined. It suggests taking an anthropomorphic lens. PSM inspired efforts in shaping pre-training priors such as Alignment Pretraining. New evidence can update PSM: agent behaviours in 2026 suggest that different training pressures create split personas that activate in different contexts. For further discussions, see thoughts on the persona selection model.
To give an example from my work which is closer to classic ML, I was recently working on testing the theory that sabotage capability has a simple latent structure that makes model performance on control evals predictable from a few parameters. Our model helped generate function classes to test, and favoured exploration into simpler function families. We found that a simple function with few parameters indeed explained our held-out data, which was evidence in support of our model. Had it been the case that thorough, principled search over simple function classes failed to predict held-out data, I would have updated against the model.
Update with uncertainty in mind
When updating predictive models based on data, error bars usually do not account for all relevant sources of uncertainty, and it’s good to keep in mind the full picture when updating.
To illustrate, say I want to learn how well a weak model detected a stronger model’s stealthy attacks in an AI control context. It turns out that the weak monitor caught 80% of attacks at a small auditing budget in LinuxArena. My prior had a mean of 50%. If I just look at the 80% number, the monitor model punches surprisingly above its weight.
At this moment, AIs are not very good at being principled about accounting for sources of uncertainty. I expect that they will become better very soon, but in the meantime, it’s worth checking whether the AI gave a reasonably complete and fair picture of the uncertainty.
Predict senior researchers’ research takes
One of the most useful exercises for improving research taste is intentionally trying to predict what senior researchers would say when discussing research.
A few ways to practice:
While trying to predict senior researchers, I often notice intellectual idiosyncrasies. It’s helpful to understand why these habits of mind are effective while being clear about their limitations. For example, Alex Cloud likes to do the MNIST-test before picking experiments:
The MNIST-test is useful because it selects for methods that work on neural networks of varying scales, not just current-day AIs. However, as Cloud notes in the post, the selection heuristic can miss many good ideas that depend on capabilities or structure absent from the MNIST setting.
Use metacognitive strategies to make predicting and updating habitual
One piece of advice I received was ‘predict the outcome of your experiment’. I think this is good advice. Pre-registering experimental predictions by using e.g. Fatebook helps me notice when I should be surprised and update my model of the world.
Remembering to predict and update requires deliberate effort initially. Ideally, we want this act to be intuitive, habitual, and agile. I find rationalist metacognitive strategies to be very helpful for building mind habits. I especially recommend people carefully read Tuning your Cognitive Strategies.
Metacognitive techniques look like doing structured introspection: noticing thought and the felt sense of thought, making increasingly granular observations about cognitive processes, and then deliberately engineering them. In general, noticing a thought pattern is the main bottleneck to refining it.
To give an example, say I am trying to predict the test set in a train/test split, and I also have available another train set that’s bigger but more off-distribution. I find that if I use this other train set, I do better on the target test set. I might notice a mild sense of surprise. At this point, I should encourage my mind to ask, why am I surprised? I want to bring my implicit model into the front stage, and think about how my model would change if this were true. Then I might want to be a bit skeptical, and run through possibilities for things I or my agent might’ve done wrong. Maybe I go double-check these. These thoughts are not heavy-duty. They should be System 1. The emphasis here is that noticing surprise should naturally lead to a whole sequence of thoughts and reactions.
Spend more effort on research communication
Communication quality bottlenecks the reception, adoption, and thus expected impact of the research. Researchers in and out of the community have emphasized this point:
There are great resources out there for writing papers and giving talks. I will focus on research updates, which are more informal than papers, talks, posters, and blog posts. Examples of research updates include weekly research slides, Slack updates, and short progress talks.
Writing good research updates helps mentors allocate their limited attention to discussions and decisions that actually matter, instead of spending most of their time getting up to speed with the project. See Fig 1 for an illustrative model.[1]
Fig 1: The mentor starts off with a bit more context than the IC (individual contributor), but the IC continuously gains more context, so it’s particularly important for the IC to get the mentor back in the game quickly and regularly with every research update. I drew a sketch and Astra made it nice.
My sense is the majority of people starting empirical safety research aren’t calibrated about the importance of good research updates and should front-load effort into improving this skill. This was true in my case and seems true after talking to my peers.
Readers of research updates are mentors, managers, collaborators, and colleagues. They are busy, so they are not caught up on the low-level details, but know a lot about research, AI safety, and the general direction. So, good research updates should be both digestible and rigorous.
To illustrate what I mean, I’ll say more about what makes good Slack update threads and plots. For advice on making good slides, see Tips On Empirical Research Slides.
Writing good Slack update threads
Good update threads vary in style, but usually they do the following really well:
At first, one way of reaching the rigorous and skimmable sweet spot for writing Slack threads is to write a rigorous research update, then spending some time making the update digestible, then combining them. By 'at first' I mean the first <5 times. I think people should expect to get way better at this very quickly. At the start of MATS, I found the idea of spending an entire day polishing my research slides excessive, but in hindsight front-loading communication level-ups would clearly have been worth it.
Making good plots
A plot is an argument in miniature and it's what people read. When I skim a paper, I usually read the title, the abstract, and then the main figures.
Most early-career empirical safety researchers should spend more time making the plots nice. Whenever I feel like I’ve put in enough effort on making my plots nice, it’s usually the case that I should’ve spent even more time.
There’s some great advice on plotting which I will not reiterate. What I do want to talk about is how I think about what makes plots easy to understand.
I’ve heard the advice that plots should be simple. It’s worth emphasizing that simple means simple relative to the existing scientific understanding. To my 10 year old self a calibration plot would have been not simple, but to the scientific community it’s well known, recognizable, and hence simple.
Plots should fit into templates. Often, bar plots say “this one is higher than this one”, diverging line plots say “this line goes up, this one doesn't”, and Pareto frontier plots usually say “we push the Pareto curve”. I give more examples in Figure 2.
Fig 2: Plots that fit existing templates are way easier to understand. I give some examples of plot templates and what they tend to communicate. The data is synthetic and meant to illustrate the main points. I made the figures with Astra.
Be calibrated about the process
Researchers frequently make judgments about what progress looks like, how much their time is worth, or whether some investment in energy or resources is worth it. I find it helpful to model these beliefs about these strategic considerations explicitly so we can form more accurate beliefs about them. I give some examples below:
Good confusion feels different from bad confusion
I have felt both good confusion and bad confusion. Good confusion reflects real uncertainty in the world; bad confusion is where I don't know what's going on in the project.
An example of good confusion: I want to find a good metric, but I don’t know what the right one is. The quantity I’m trying to estimate is conceptually fuzzy, and I don’t even know if it’s a natural quantity in the world. I need to try a bunch of stuff to slowly clarify if this quantity is well-defined, existent, and natural.
An example of bad confusion: I don’t know if I’m in the right stage to wrap up the project, or if I should continue exploring more directions within the paper, and I’m doing both at the same time while not knowing what I should expect to have done in two weeks.
I find that the felt sense is pretty obviously different for both, so it’s useful to develop an intuitive handle on what state I’m in, and then practice being comfortable with good confusion while reducing bad confusion.
Sometimes it’s better to burn money than time
I find it helpful to estimate the expected value of a particular task in the next few hours / days / weeks, and then ask if the appropriate amount of money and time is being used. For example, at both MATS and Redwood, there were times where I erred on the side of using too little money while spending way too much time on things that don't matter. In those cases, I probably should have traded time for money by, for example, using Modal instead of a cheaper GPU provider because it fanned out faster, or simply running more samples of things because higher sample size was higher information value in that scenario.
Small time commitments become costly in attention
The impact of empirical projects is heavy-tailed–a very good project is way more useful than a good project. Hence, protecting focus matters a lot.
When focusing on one project, I find that small work-related time commitments such as an extra 1-2 hours of meeting a week become way more costly in attention. If I take 3 meetings a day, then taking 4 meetings does not change the type of work I do. If I normally take no meetings, then taking a meeting will fragment my day, which disrupts focus. This Paul Graham post makes the full point: Maker's Schedule, Manager's Schedule.
Know which level of ‘being done’ to aim for
Projects with different levels of completion requirements can demand an order of magnitude more or less effort. I think researchers should be clear about the level of ‘done’ they want to achieve and plan accordingly.[2]
Here's a frame I find useful:
My sense is that a paper-complete project is 1/2-1 OOM higher effort compared to an epistemic-complete project. Take Alignment Faking as an example. Ryan Greenblatt demonstrated the core results, and it took a much larger team, mostly of Anthropic researchers, for the project to reach a paper-complete state.
There are merits to choosing different levels of ‘done’. Epistemic-complete projects finish way faster, while good papers reach people beyond the safety community, and are good for careers or visa applications. People who want to be done after epistemic-completeness should resist the urge to explore more directions after they are convinced of the core results; people who want to write good papers should expect the process to take months, even after the core results have been established.
Closing Thoughts
To practice these skills, I highly recommend assigning a generous fraction of time to writing things down.[3] I kept an informal running list of research lessons in Obsidian, which helped me consolidate what I learned. I wrote Google Docs about my confusions, where I think the project is going, and where my current model stands. I wish I’d written even more.
Most of the lessons I shared are downstream of treating technical AI safety research as a science: developing a model of the world, making predictions, empirically testing these predictions, and then updating. Having a wrong model is better than having an unfalsifiable one. In the words of Francis Bacon: “Truth emerges more readily from error than from confusion.”
This model is illustrative, and adopts a narrow take on ‘context’: at all points, mentors will in all likelihood have good object-level takes that the IC doesn’t due to having better research taste, but this sketch doesn’t account for that.
Hofstadter's law: It always takes longer than you expect, even when you take into account Hofstadter's law. Source: Wikipedia
I often start my writing session using this app, which deletes all my words if I stop typing, to get the momentum going