AI disclosure: I made detailed notes on the basis of which Claude generated a first draft that I then substantially edited.
Confidence: Written relatively quickly, unsystematically, and with many somewhat vague claims. I expect that I would change my mind about some of these claims when working them out in detail.
Edward James Young’s post Alignment engineering vs. misalignment science distinguishes two styles of alignment research. Alignment Engineering starts from a specific problem – sometimes measured by a number – and aims to develop interventions that help with the problem. Misalignment Science aims at deeper understanding and is not optimised for immediate application.
I think this distinction is very helpful. It is especially useful because it marks out an intermediate level. Debates about research strategy sometimes contrast prosaic engineering on current models with general theories that are highly mathematical or otherwise far removed from empirical work and should apply to a very wide range of systems, perhaps all intelligent agents. Misalignment Science sits in between: it may consist substantially in empirical work on current systems, but is aimed at explanation rather than immediate practical problem-solving.
Below I discuss four things: whether the benefits the post discusses are really closely tied to science-style work, why the distinction is more useful as a description of a mindset than as a way to classify research projects or areas, which criteria I would use instead to evaluate research, and what I take to be the deeper deficit of alignment research the post points to. None of the points necessarily disagree with Young’s post.
Are the posited connections as tight as they seem?
The post ties the science–engineering distinction to several benefits of science-style work. I think the connections are real but looser than suggested. My assessment of three important putative benefits of Misalignment Science:
Less capability improvement harm. Partly true. Understanding-focused work does not aim at a specific problem AI labs face, so it is less likely to remove a bottleneck to further capability improvement. But gaining a better theoretical understanding of model (mis)behaviour can lead to capability improvements downstream – either independently or by fixing specific alignment challenges.
Less risk-obscurement harm. Largely true. When Alignment Engineering optimises a quantitative metric, it is exposed to Goodhart’s law: the number moves while the underlying problem persists or becomes harder to see. Visible failures that could have served as warning shots can get suppressed. Work that aims at explanation does not optimise a target in this way and is less exposed. But to some degree, explanations can obscure risks too. Suppose a worrying behaviour, such as alignment faking, is explained as the model merely role-playing a persona. If this explanation is compelling but wrong, it might make the behaviour look less dangerous than it is, just as a gamed metric would. Interpretability illusions (Makelov et al. 2024) show how easily seemingly compelling mechanistic explanations can mislead.
Scales to higher capability levels. It depends. Understanding scales only if its underlying theories are general enough. A theory of how AI agents generally acquire and pursue goals would carry over to more capable systems. A detailed explanatory account of one feature in one model on one task may tell us little about the next generation. Being science-style is no guarantee here; what matters is the generality of the resulting theory.
A mindset rather than a way to sort projects
These observations suggest we should doubt that the distinction is very useful for evaluating concrete research projects. Three further observations buttress my scepticism:
Coarse-grained research approaches can be pursued in a science or an engineering way. Mechanistic interpretability can aim at deeply explaining how a model represents and processes something, or at building a probe that moves a detection score. Searching for misalignment incidents can aim at uncovering new theoretically significant phenomena, or at assembling a benchmark to be hill-climbed. Knowing the general kind of research project therefore settles little regarding whether it’s science or engineering.
Paradigmatic engineering work can provide theoretically useful data. An account of “what works”, and of what does not, constrains theorising. If an intervention reliably removes a behaviour across models, that is data that can inform theorising.
Work that looks for causes can be deep and theory-based, or not. Testing whether some component removes a particular failure in a particular model is causal work, but it can be closer to engineering than to Misalignment Science.
None of this implies that all research directions are equally promising. But deciding which ones are requires a detailed object-level discussion of a project’s specific potential for harm and benefit. The engineering vs. science label does not do that work.
What remains is a difference in mindset: whether a researcher searches for deep underlying explanations of (mis)alignment and treats results as something that primarily serves to help this search, or whether they treat alignment as a set of practical problems and results as indications of whether such a problem is solved. That is a useful thing to ask oneself. It is also easy to fool oneself about. Many researchers might sincerely describe their own work as building understanding, because there are no clear criteria suggesting otherwise.
A checklist for evaluating research
For evaluating research ideas and topics, rather than letting the engineering vs. science labels carry weight, I think we should try to use a checklist of the outcomes we actually care about and assess each project on each criterion separately. Examples are:
Criterion
Question to ask about a project
Capability improvement harm
Does the research contribute towards improving dangerous AI capabilities?
Risk-obscurement harm
Does it hide problems, for example by minimising the visible failures that could serve as warning shots?
Making risks or research gaps visible
Does it show where current methods fail or what we do not yet understand?
Helping automate alignment research
Does it make it more likely that automated alignment research succeeds?
Helping align superintelligence directly
Does it contribute to aligning systems far more capable than today’s?
These criteria are largely independent of the science–engineering distinction. Per the previous considerations, a science-style project can do badly on them, for instance if it builds on theories that help speed up capabilities but do not generalise well to more complex systems. An engineering project can do well, for instance by providing techniques that promise to make automated alignment research more reliable, but are not useful for current capabilities.
The underlying deficit: missing mid-level theory
The central deficit the call for Misalignment Science boils down to, in my view, is the lack of systematic theory development and testing in AI alignment research. Theories, in a broad sense, are what understanding-focused science works with. Moving alignment towards a science-based model would mean developing more and better theories and differentially testing them.
To make progress here, I think we should systematically map what kinds of theories alignment research has and which kinds it needs. It has the following:
- very specific mechanistic accounts of particular representations and processes in individual AI systems on particular tasks (e.g., presented in individual mechanistic interpretability papers);
- general mathematical models, for example from decision theory;
- a few general frameworks that do not really make specific testable predictions, and instead can be used as a lens to understand most kinds of model behaviour (e.g. the persona selection model).
My conjecture is that what is largely missing are mid-level theories: mechanistic models of alignment-relevant properties that are concrete enough to predict specific results and thus be testable, but general enough to hold across some models, tasks and measures.
Mechanistic interpretability results that are true of many different kinds of models may go in this direction, e.g. Arditi et al. (2024) find that refusal is mediated by a single direction in the activations of 13 open-source chat models, and Chen et al. (2025) identify “persona vectors” that can be used to monitor and steer traits such as sycophancy, and to predict how finetuning shifts them. But they are empirical findings, they are not by themselves theoretical accounts. Moreover, they are isolated – they do not amount to explanations of how alignment-relevant properties such as LLM motivation and values are structured and how their components interact to produce behaviour. They are better seen as building blocks for mid-level theories than as such theories themselves.
Philosophy of neuroscience offers useful vocabulary here. Craver (2006) places mechanistic models on a continuum of completeness: mechanism sketches contain gaps, marked by black boxes or masked by filler terms; schemata abstract away from the details of particular mechanisms; ideally complete descriptions specify all relevant parts, activities and their organisation. For alignment-relevant properties such as LLM motivation and values, even very incomplete models would be progress, as long as they make some specific predictions and can be generalised across some systems and contexts.
AI disclosure: I made detailed notes on the basis of which Claude generated a first draft that I then substantially edited.
Confidence: Written relatively quickly, unsystematically, and with many somewhat vague claims. I expect that I would change my mind about some of these claims when working them out in detail.
Edward James Young’s post Alignment engineering vs. misalignment science distinguishes two styles of alignment research. Alignment Engineering starts from a specific problem – sometimes measured by a number – and aims to develop interventions that help with the problem. Misalignment Science aims at deeper understanding and is not optimised for immediate application.
I think this distinction is very helpful. It is especially useful because it marks out an intermediate level. Debates about research strategy sometimes contrast prosaic engineering on current models with general theories that are highly mathematical or otherwise far removed from empirical work and should apply to a very wide range of systems, perhaps all intelligent agents. Misalignment Science sits in between: it may consist substantially in empirical work on current systems, but is aimed at explanation rather than immediate practical problem-solving.
Below I discuss four things: whether the benefits the post discusses are really closely tied to science-style work, why the distinction is more useful as a description of a mindset than as a way to classify research projects or areas, which criteria I would use instead to evaluate research, and what I take to be the deeper deficit of alignment research the post points to. None of the points necessarily disagree with Young’s post.
Are the posited connections as tight as they seem?
The post ties the science–engineering distinction to several benefits of science-style work. I think the connections are real but looser than suggested. My assessment of three important putative benefits of Misalignment Science:
A mindset rather than a way to sort projects
These observations suggest we should doubt that the distinction is very useful for evaluating concrete research projects. Three further observations buttress my scepticism:
None of this implies that all research directions are equally promising. But deciding which ones are requires a detailed object-level discussion of a project’s specific potential for harm and benefit. The engineering vs. science label does not do that work.
What remains is a difference in mindset: whether a researcher searches for deep underlying explanations of (mis)alignment and treats results as something that primarily serves to help this search, or whether they treat alignment as a set of practical problems and results as indications of whether such a problem is solved. That is a useful thing to ask oneself. It is also easy to fool oneself about. Many researchers might sincerely describe their own work as building understanding, because there are no clear criteria suggesting otherwise.
A checklist for evaluating research
For evaluating research ideas and topics, rather than letting the engineering vs. science labels carry weight, I think we should try to use a checklist of the outcomes we actually care about and assess each project on each criterion separately. Examples are:
Criterion
Question to ask about a project
Capability improvement harm
Does the research contribute towards improving dangerous AI capabilities?
Risk-obscurement harm
Does it hide problems, for example by minimising the visible failures that could serve as warning shots?
Making risks or research gaps visible
Does it show where current methods fail or what we do not yet understand?
Helping automate alignment research
Does it make it more likely that automated alignment research succeeds?
Helping align superintelligence directly
Does it contribute to aligning systems far more capable than today’s?
These criteria are largely independent of the science–engineering distinction. Per the previous considerations, a science-style project can do badly on them, for instance if it builds on theories that help speed up capabilities but do not generalise well to more complex systems. An engineering project can do well, for instance by providing techniques that promise to make automated alignment research more reliable, but are not useful for current capabilities.
The underlying deficit: missing mid-level theory
The central deficit the call for Misalignment Science boils down to, in my view, is the lack of systematic theory development and testing in AI alignment research. Theories, in a broad sense, are what understanding-focused science works with. Moving alignment towards a science-based model would mean developing more and better theories and differentially testing them.
To make progress here, I think we should systematically map what kinds of theories alignment research has and which kinds it needs. It has the following:
- very specific mechanistic accounts of particular representations and processes in individual AI systems on particular tasks (e.g., presented in individual mechanistic interpretability papers);
- general mathematical models, for example from decision theory;
- a few general frameworks that do not really make specific testable predictions, and instead can be used as a lens to understand most kinds of model behaviour (e.g. the persona selection model).
My conjecture is that what is largely missing are mid-level theories: mechanistic models of alignment-relevant properties that are concrete enough to predict specific results and thus be testable, but general enough to hold across some models, tasks and measures.
Mechanistic interpretability results that are true of many different kinds of models may go in this direction, e.g. Arditi et al. (2024) find that refusal is mediated by a single direction in the activations of 13 open-source chat models, and Chen et al. (2025) identify “persona vectors” that can be used to monitor and steer traits such as sycophancy, and to predict how finetuning shifts them. But they are empirical findings, they are not by themselves theoretical accounts. Moreover, they are isolated – they do not amount to explanations of how alignment-relevant properties such as LLM motivation and values are structured and how their components interact to produce behaviour. They are better seen as building blocks for mid-level theories than as such theories themselves.
Philosophy of neuroscience offers useful vocabulary here. Craver (2006) places mechanistic models on a continuum of completeness: mechanism sketches contain gaps, marked by black boxes or masked by filler terms; schemata abstract away from the details of particular mechanisms; ideally complete descriptions specify all relevant parts, activities and their organisation. For alignment-relevant properties such as LLM motivation and values, even very incomplete models would be progress, as long as they make some specific predictions and can be generalised across some systems and contexts.
References