TL;DR:

Inoculation prompting and inoculation adapters have received increasing attention recently as a promising approach for midtraining interventions, reducing reward hacking and misalignment in general.

I share some thoughts on the promises and pitfalls of the approach, connections to unlearning, SLT and functional sparse decompositions as well as potential extensions and open questions below.

Some experiments that directly arose from ideas presented in this post are covered in this post.

Learning paradigms

There are two basic principles most learning in LLMs relies on: explicit parametric learning (we are tweaking the weights of the model to minimize some optimization target) and in-context learning (when we are relying on some inner optimization and inductive learning for the model to perform).

Essentially, any LLM, similar to almost any deep learning model is just a function of two variables:

, where is some input (context) space, and is some weight space.

I'll use to denote a feature or a concept, and stick to this notation below as well. Although we could argue that any piece of information (e.g. all the contents of Lord of the Rings saga) could potentially be compressed and represented via a certain , I'll primarily rely on much more compressible notions, like "color", "shape" or "style"^[1].

Strip down embeddings and tokenization^[2] and you are left with just a mapping between two spaces, both of which you could potentially optimize over.

Here's the sample table of the problem:

Both modify the way the model acts upon the input, and differ just in the way modification is applied. For transformer-based models, ICL can only act through the attention, while weight space modifications allow arbitrary changes to circuitry of the model itself.

Recently, Inoculation prompting and inoculation adapters were introduced as techniques to mitigate emergent misalignment and allow learning from data, where benign features might appear entangled with harmful ones, and avoid extensive data filtering. In essence, both rely on specifying "semantic representation" of the undesired trait, that's supposed to be learned from the data distribution, besides useful ones, to be present at training time, so there's no need for the model to learn it internally.

Although it captures some part of the story, I find it slightly incomplete and misleading. Some thoughts and experiments on the subject are below.

Thoughts on inoculation

Xu and Strohmer (2025) did an amazing job formalizing both, feature and data-point unlearning.

Essentially, unlearning seeks to solve the following problem:

Given the data , an unlearning variable , and a target variable , how can we construct a released outcome (model) that exposes minimal information about while preserving as much task-relevant information about as possible?

Another way to see it is from the perspective of utility-unlearning trade-off (similar to the accuracy-robustness tradeoff (Wu et al., 2023) in adversarial learning):

where is the function from some model class defined by the architecture, is an arbitrary utility function^[3] and denotes a mutual information between distributions of , the model on which unlearning is evaluated^[4] and unlearning target .

The authors additionally separate feature unlearning and marginal data-point unlearning:

• Feature unlearning

No observer should be able to infer anything about from the model's outputs^[5] beyond prior knowledge.

For example, if you are trying to unlearn information about the day of the week a person met person , and there's no other information, an unlearned model should assign a uniform probability to any day, given the query. Although underestimation of the probability of the target answer makes it unlikely for it to be sampled from the model, it still provides some information to the potential adversary, unlike the uniform measure.

• Marginal data-point unlearning

No observer should be able to infer anything about the marginal information contributed by the unlearn set (beyond the retain set) from the model’s outputs, beyond prior knowledge.

You can possibly think of as a tabular data matrix, where rows correspond to individual samples, while columns correspond to features. "Feature unlearning" aims to forget a certain feature column which attributes correspond to the "retain" set. However, the approach is non-restrictive, and we might aim to forget an association between one feature and another feature .

Contrary to it, "marginal data-point unlearning" aims to forget a subset of rows entirely, which makes it closer to how common unlearning approaches are formalized and evaluated.

Obviously, similarly to almost all natural observables, the distinction between features and data-points is not discrete and should be though in a continual spectral manner.

Another view can be presented by the weight/anchor unlearning:

No observer should be able to distinguish the released model from the model that would have been obtained by training from scratch on the retain set alone.

I highlight that the first two unlearning "principles" demanded some properties from distributions in the activation space, while the last one is about distributions on the weight space.

However, due to computational infeasibility of statistical inference and hypothesis testing in the parameter space, we often restrict unlearning to the activation space formalism, similarly to the membership inference approaches (Carlini et al., 2022).

Now let's return to inoculation and take a fresh look at the subject:

• Initially, inoculation prompting appeared as a tool to suppress some undesired behavior (Tan et al., 2026, Wichers et al., 2025) by conditioning its expression on a human-written lexical context that encodes in some way, while learning on a joint distribution from which both and a desired behavioral feature might be inferred. Although there were initial attempts to connect it to unlearning, I see no reason, either theoretical or empirical, to believe that inoculation results in unlearning of any latent traits rather than their suppression. Given the compression necessary to build a sufficient world model from billions of data points, I highly doubt that genuine unlearning is possible in any foundation models. Recent works that study membership inference, and its connections to memorization and generalization in language models point towards the same conclusion. Similar argument regarding conditionalization has been made in this blogpost by Riche & nielsrolf.
  
• Formally, inoculation as a sequential learning problem. At first, a probabilistic model is being learnt, while is being used at inference time, which, ideally, should increase the expression of , while suppressing an expression of .
  However, an initial approach was found to allow multiple weird backdoors (Dubinsky et al., 2026), display inconsistent transferability to more difficult domains and trait combinations, while being sensitive to the instruction following abilities of the model as well as the specific instruction specified (Riche et al., 2026).
• You might feel like the inoculation shares a lot with the feature unlearning task, and you are probably right. The task is still identical, at least regarding "unlearned" feature , while we have an additional objective to encode in a way that's functionally independent from even though they are leaned from the same data.
  
  No observer should be able to infer anything about from the model's outputs beyond prior knowledge, besides the marginal information about contained in .
  
  A trivial, but potentially important counterexample can be imagined by taking , which renders any selective inoculation impossible. You can imagine a gaussian mixture separation problem with two gaussians sharing the the same mean for an intuitive model.
  

• Inoculation adapters (Riche et al., 2026) build on the same idea, but rely on learnt inoculation and explicit split between inoculation and conditionalization steps instead. While prompting relies on mutual information between features and being inferred in-context from the human-written prompt and dataset and their approximate minimization during training, inoculation adapters learn a certain behavioral pattern (or, essentially, a trait-associated steering / task vector ) that expresses a certain concept unconditionally instead.
  
  The process is two-staged:
  
  Train a low-rank perturbation of the model . Given "clean" data distribution with undesired feature encoded within some natural language contexts and "optimizer" function that takes data and initialization as parameters, we obtain that should express unconditionally.
  
  At the second stage we train on a mix of and , hoping that all the information that can be inferred about has already been encoded by , and results in an encoding of that should presumably be independent from the encoding of made previously.
  

The key problem is, potentially, the fact that the point was not obtained using any gradient based learning, and we never controlled, either via trust region or any other gradient-aware method, whether the generalization properties on it will be preserved. Instead, we relied on some approximate weight space arithmetic, and it magically worked.

• An approach resembles similar research performed regarding task vectors or task arithmetic by Ilharco et al. (2023) who showed that multi-task performance can be composed as a weighted combination of task vectors in the parameter space. These are fine-tuned from the same checkpoint using different tasks to produce a model , and then summing the difference vectors into , the model with multi-task capabilities.
  
  Most of the time the task arithmetic property is defined the following way (Ortiz-Jimenez et al. (2023)):
  
  Consider a set of task vectors with associated non-intersecting task supports . Then the network , parametrized by satisfies the task arithmetic property around with respect to and if:
  
  
  
  Essentially, it means that adding doesn't modify the distribution parametrized by outside the "region", specified by . Although such behavior could be attributed to linearity of a form , it's not a unique explanation.

• However, inoculation adapters present a somewhat distinct pattern. Firstly, they don't use full-finetuning, which was the case for the task arithmetic, but use LoRA adapters with increasingly small rank instead, which appears to be crucial.
  
  Secondly, they show that to a certain degree, the task arithmetic appears to hold not only at , but also in some neighborhood of it. Somewhat similar effects have been studied in Ortiz-Jimenez et al. (2023) and extended in Zhou et al. (2024), who showed that it's possible to interpolate between several checkpoints fine-tuned from the same base model, which results in approximately linear interpolation between their features.
  
  Instead of assuming that the function of interest is linear in the neighborhood of , we could assume it to be a sparse linear combination of context dependent functions.
  
  Ortiz-Jimenez et al. (2023) claimed that such property, known as weight disentanglement, emerges during pretraining and manifests with distinct directions in the weight space governing separate regions in the function space of a form:
  
  
  where for and and for
  
  The important point is that task arithmetic does not require linearity in weight space. It only requires a form of functional modularity/weight disentanglement. In essence, this is another instance, where a broader notion of sparse compositionality (Danhofer et al., 2025) might play its role. If computations are indeed given by some sparse functional composition, this might explain both low trait interference and our ability to steer it conditionally.
  
• Another way to see it is from the lens of recent advancements in SLT (singular learning theory). Although I find it less suitable to explain the large scale learning dynamics during any stages of pre- or post-training, as it mostly captures the behavior at the end of training, or near the current parameter value, it might appear to be increasingly more useful for training time interventions.
  
  If the model, conditional on the set of contexts , is degenerate along the direction , while not being degenerate conditional on the task , and changing the notation from and , effects observed in inoculation adapters might appear to be slightly less surprising.
  
  Potentially, we could argue that inoculation adapters change the local statistical model being optimized by conditioning the task loss on a frozen parameter component that already implements the undesired trait. On the joint desired + undesired training distribution, this can render task-adapter directions associated with the undesired trait locally non-identifiable or nearly degenerate, while leaving directions responsible for the desired task identifiable.
  
  Informally speaking, removing the inoculation adapter changes the conditional loss geometry again, but the learned task adapter has already been optimized in a geometry in which the undesired direction carried little optimization pressure.
  
  Given that one could think of low-rank adapters as the ones for which there exists a basis in which the movement in the parameter space is restricted to only components, this explanation appears quite plausible.

• Overall, I still find it quite surprising that the same identified arithmetic holds not only at , but at other points in the neighborhood of as well (as if the functions were translation invariant and hold such strange symmetry in the weight space), e.g. at .
  
  I expect the joint effect of local linearity (which corresponds to dense low-rank structures) and sparsity (which corresponds to high rank, but entry-wise structures) to be the closest model. This work by Bertsimas et al. (2023) could serve as a good introduction for anyone interested.
  
  

Hypotheses:

• Traits/features can be encoded as directions or subspaces in the residual stream.

E.g. let where , and can be represented as a rank-1 adapter. Then, any output feature is given before non-linearity by . As is a scalar, the mapping corresponds to , a linear steering with a data-dependent scale.
Similar argument can be applied to higher rank adapters with and without major changes, yet now steering is applied along the subspace spanned by the columns of instead of one-dimensional vector.

Thus, any low-rank adapters restrict the search space and provide an essential inductive bias. As low-rank matrices have measure zero in the space of , it's almost impossible for unconstrained gradient-based methods to find these precisely, and it's the combination of low-rank mappings and well-defined compressible traits that drives the success of inoculation.

• Information contents from individual data-points and factual knowledge are highly distributed and entangled. I do not expect them be represented via (at least, dominant and firstly learned) subspaces in the residual stream. Thus, inoculation can hardly work for general unlearning tasks, such as TOFU (Maini et al., 2024) or WMDP (Li et al., 2024).
  Similar arguments were independently made for model editing (Hase et al., 2023) and unlearning (Lee et al., 2025), showing that in both cases localization barely predicts where edits work.
  
• In-context learning is much more noisy and should struggle to make trait feature encoding stable enough as it's often overwritten by attention maps.
  
  Inoculation adapters are applied to each token via a similar mapping , while learnt prompts or prefixes act through the KV-cache via , where learned contents in general correspond to only few first ones. It might be harder to do error-correction and preserve a stable trait direction in the residual stream through self-attention than linearly via MLP writing directly into the residual stream.
  

Possible extensions I'd be interested in:

I intend to update this list in the future.

• Extend the task arithmetic theory to low-rank adapters. Although, as a concept, task vectors are well-known and LoRA merging is used widely in production model editing -- especially in Video, Image and multimodal domains, a theory under which conditions is it possible and why it happens is lacking.
  
  The tangent space of LoRA is degenerate at initialization, and it's possible to reparametrize the weights to make the any initialization viable:
  
  
  
  I expect non-trivial connections with SLT to emerge here, providing some insights into the feasibility of inoculation and predictability of its success over different training stages.
  
• Essentially, as we've seen, inoculation differs from feature unlearning in only one tweak. The first one is trying to minimize the mutual information between the forget set and outputs distribution and approximately preserving the retain set -- which I assume is ultimately infeasible without major tradeoffs^[6]. The second one gives up on it and just makes the knowledge or feature expression conditional on the trigger, that's potentially narrow.
  
  There are certain information theoretical limits on its compression as well (at least in finite precision arithmetic). It might be reasonable to borrow techniques that have been developed there over the years of research and apply them to the problem. Ryuto et al., (2025) did exactly the same to transfer techniques between membership inference attacks and human-machine text detection and built a good library out of it.
  
• Brown et al. (2026) studied how optimizers impose inductive biases on learning dynamics and spectral properties of weight matrices (LoRA adapters in particular), showing that the rate of emergent misalignement is negatively correlated with statistics, such as stable rank and efficient rank, i.e. the flatness of the spectra. In other words, emergent misalignment seems to be encoded in a very narrow subspace, while biasing learning from towards more uniform spectra seems to prevent misalignment. I think one of the baselines is still lacking. OFT, GSOFT and other manifold optimization methods with orthogonal constraints preserve the uniform spectra by design, and this should be a study of rank / subspace encoding constraints rather than of optimizers themselves. Similar results on inductive biases of -based optimizers were observed e.g. in Liu et al., (2026), yet the list is nowhere near exhaustive. Do orthogonality constrained methods allow fine-tuning without any emergent misalignment, which should motivate their increased use for safety purposes, or is there some "optimal" intermediate spectral distribution that prevent misalignment better?
  
  Essentially, if certain concepts have an inductive bias towards being encoded in a low-rank way (e.g. via a vector), having an opposing force or restricting the effective rank of the update might complicate the search for such encoding.
  
  Experiment: use OFT or any similar approach to fine-tune the model on a task where emergent misalignment emerges most of the time. Does is happen with OFT? Does it happen at similar number of steps or does orthogonality postpone an emergence?

===

If you found this work helpful and would like to cite it, please use:

@misc{kudriashov2026reflections,
title = {Reflections on inoculation adapters},
author = {Kudriashov Sergei},
year = {2026},
howpublished = {Independent},
note = {Blogpost.  Available on LessWrong forum.},
url = {https://www.lesswrong.com/posts/GnG2ono8vdoJvyF9f/reflections-on-unlearning-and-inoculation}
}

Acknowledgements

Thanks to Samuel Ratnam and Maxime Riche for discussions and valuable comments on this post.

1. ^{^}

   Whether concepts such as "deception" or "reward seeking" are similarly compressible in the whole generality is an open question. We could argue that an existence of a lexical instance for them is already a significant argument that they are. Yet "all the content of the Lord of the Rings" saga is a great compression as well. In fact, this discussion is somewhat linked to the world-famous in narrow circles "Big Number Duel" at MIT. See here and here for references. ↩︎

2. ^{^}

   Which is what most theoretical in-context learning studies do anyway, see Ahn et al. (2023) and Collins et al. (2024) for reference.

3. ^{^}

   It can be a loss function, some benchmark performance or anything along these lines. ↩︎

4. ^{^}

   With, most commonly, (S' = S). ↩︎

5. ^{^}

   In general, we can treat any intermediate hidden state or residual stream representation as a model's output as well, depending on the level of generality we want to operate on. ↩︎

6. ^{^}

   I think it's reasonable to assume, given extensive prior research, that most interesting features -- especially the ones we care about, such as deceptiveness, situational awareness, latent behavioral traits of various personas -- are weirdly entangled and can hardly be disjointed as easy as e.g. ability to write ALL CAPS. ↩︎

7. ^{^}

   Which is closer to the MLP-Mixer and other models, that can be viewed as kronecker-factorized nonlinear operators.

8. ^{^}

   I don't think it's correct to identify in-context learning with mesa-optimization to full extent, despite clear parallels. and evidence to e.g. perform gradient descent in the latent space. At least it seems reasonable to assume that not every computational circuit is necessarily an optimization routine or is necessarily serving as its subpart.

9. ^{^}

   I'm naming these two just due to personal familiarity. ↩︎