# Title: Reflections on unlearning and inoculation Post ID: `GnG2ono8vdoJvyF9f` Version: `draft` Context for LLMs/AI Agents: This is a markdown translation of a draft post. You probably got here because a user shared a link to this page with you. We built this feature to help users get feedback on their posts, and to make it easier for AI agents to help users with their posts. As part of the feature, we also provide API endpoints for leaving inline comments/suggestions/etc on the post. The API endpoints are documented in the "Helping Users With Drafts" section of the Markdown API documentation. The content of the post is below, between the two horizontal rules. There may be additional horizontal rules in the post content. To help disambiguate, the post content should be followed by a "Comment Threads" section if the post has any open comment threads, and then a "Navigation" section; neither is part of the post. * * * **TL;DR**: ---------- *Inoculation prompting and inoculation adapters have received increasing attention recently as a promising approach for* [*midtraining interventions*](https://arxiv.org/html/2609.15886v1), [*reducing reward hacking*](https://www.anthropic.com/research/emergent-misalignment-reward-hacking) *and* [*misalignment*](https://arxiv.org/pdf/2606.30252) in general. *I share some thoughts on the promises and pitfalls of the approach, connections to unlearning, SLT and functional sparse decompositions as well as potential extensions and open questions below.* *Some experiments that directly arose from ideas presented in this post are covered in* [*this post*](https://www.lesswrong.com/posts/8AYQrvD4HEh8MEjgR/evaluating-unlearning-and-inoculation-task-vector). Learning paradigms ------------------------ There are two basic principles most learning in LLMs relies on: explicit parametric learning (*we are tweaking the weights of the model to minimize some optimization target*) and in-context learning (*when we are relying on some* *inner optimization* *and inductive learning for the model to perform*). Essentially, any LLM, similar to almost any deep learning model is just a function of two variables: $F(X,W)$, where $X \in \mathcal{X}$ is some input (context) space, and $W \in \mathcal{W}$ is some weight space. I'll use $Z$ to denote a feature or a concept, and stick to this notation below as well. Although we could argue that any piece of information (e.g. all the contents of Lord of the Rings saga) could potentially be compressed and represented via a certain $Z$, I'll primarily rely on much more compressible notions, like "*color*", "*shape*" or "*style*"[^l32nh4yc3qk]. Strip down embeddings and tokenization[^dgrgjjsjf9] and you are left with just a mapping between two spaces, both of which you could potentially optimize over. Here's the sample table of the problem: | Sampling distribution | Context (ICL), $P_W(X\\|Z)$ | Weights, $P_{W+ \\Delta W}(X)$ | | --- | --- | --- | | Human-specified | Any prompt | *Idk, if you have a good prior, sample the right weights and solve alignment finally* | | Learnt | [Prefix-tuning](https://arxiv.org/abs/2101.00190), [prompt-tuning](https://arxiv.org/pdf/2507.06085v2), [prompt optimization](https://arxiv.org/pdf/2502.16923) | [LoRA](https://arxiv.org/abs/2106.09685), [OFT](https://openreview.net/forum?id=K30wTdIIYc), [IA3](https://arxiv.org/abs/2205.05638), [BitFit](https://arxiv.org/abs/2106.10199) etc. | Both modify the way the model acts upon the input, and differ just in the way modification is applied. For transformer-based models, ICL can only act through the attention, while weight space modifications allow arbitrary changes to circuitry of the model itself. Recently, I[noculation prompting](https://www.lesswrong.com/posts/AXRHzCPMv6ywCxCFp/inoculation-prompting-instructing-models-to-misbehave-at) and [inoculation adapters](https://www.lesswrong.com/posts/qd3qhxgEmQAXR2ZK5/inoculation-adapters-improve-upon-inoculation-prompting) were introduced as techniques to mitigate [emergent misalignment](https://arxiv.org/abs/2502.17424) and allow learning from data, where benign features might appear entangled with harmful ones, and avoid extensive data filtering. In essence, both rely on specifying "semantic representation" of the undesired trait, that's supposed to be learned from the data distribution, besides useful ones, to be present at training time, so there's no need for the model to learn it internally. *Although it captures some part of the story, I find it slightly incomplete and misleading. Some thoughts and experiments on the subject are below.* Thoughts on inoculation ======================= [Xu and Strohmer (2025)](https://www.math.ucdavis.edu/~strohmer/papers/2025/Machine_Unlearning.pdf) did an amazing job formalizing both, feature and data-point unlearning. Essentially, unlearning seeks to solve the following problem: > *Given the data* $X$*, an unlearning variable* $Z$*, and a target variable* $Y$, how can we construct a released outcome (***model***) $S(X,Y,Z)$ *that exposes minimal information about* $Z$*while preserving as much task-relevant information about* $Y$*as possible?* Another way to see it is from the perspective of utility-unlearning trade-off (similar to the accuracy-robustness tradeoff ([Wu et al., 2023](https://openreview.net/forum?id=bl9zYxOVwa)) in adversarial learning): $\inf_{f \in \mathcal{H}} \qquad (1 - \lambda) \underbrace{\mathcal{C}(Y; S, Z)}_{\text{utility/capability preservation}} + \lambda \underbrace{I(S'; Z)}_{\text{unlearning}}$ where $f$ is the function from some model class $\mathcal{H}$ defined by the architecture, $\mathcal{C}(\cdot)$ is an arbitrary utility function[^h2hda8bae3l] and $I(\cdot; \cdot)$ denotes a mutual information between distributions of $S' = f(X,Z)$, the model on which unlearning is evaluated[^31tni99gdpi] and unlearning target $Z$. The authors additionally separate **feature unlearning** and **marginal data-point unlearning:** * **Feature unlearning** > *No observer should be able to infer anything about* $Z$ *from the model's outputs*[^3lgkdnxuiz3] *beyond prior knowledge.* For example, if you are trying to unlearn information about the day of the week a person $A$ met person $B$, and there's no other information, an unlearned model should assign a uniform probability $1/7$ to any day, given the query. Although underestimation of the probability of the target answer makes it unlikely for it to be sampled from the model, it still provides some information to the potential adversary, unlike the uniform measure. * **Marginal data-point unlearning** > No observer should be able to infer anything about the marginal information contributed by the unlearn set (beyond the retain set) from the model’s outputs, beyond prior knowledge. You can possibly think of $X$ as a tabular data matrix, where rows correspond to individual samples, while columns correspond to features. "*Feature unlearning*" aims to forget a certain feature column which attributes correspond to the "*retain*" set. However, the approach is non-restrictive, and we might aim to forget an association between one feature $W$ and another feature $Z$. Contrary to it, *"marginal data-point unlearning"* aims to forget a subset of rows entirely, which makes it closer to how common unlearning approaches are formalized and evaluated. Obviously, similarly to almost all natural observables, the distinction between features and data-points is not discrete and should be though in a continual spectral manner. Another view can be presented by the **weight/anchor unlearning**: > *No observer should be able to distinguish the released model from the model that would have been obtained by training from scratch on the retain set alone.* I highlight that the first two unlearning "principles" demanded some properties from distributions in the activation space, while the last one is about distributions on the weight space. However, due to computational infeasibility of statistical inference and hypothesis testing in the parameter space, we often restrict unlearning to the activation space formalism, similarly to the membership inference approaches ([Carlini et al., 2022](https://arxiv.org/pdf/2112.03570)). Now let's return to inoculation and take a fresh look at the subject: * Initially, inoculation prompting appeared as a tool to suppress some undesired behavior $Z$([Tan et al., 2026](https://arxiv.org/abs/2510.04340), [Wichers et al., 2025](https://arxiv.org/pdf/2510.05024)) by conditioning its expression on a human-written lexical context that encodes $Z$ in some way, while learning on a joint distribution from which both $Z$ and a desired behavioral feature $Y$ might be inferred. Although there were initial attempts to connect it to unlearning, I see no reason, either theoretical or empirical, to believe that inoculation results in unlearning of any latent traits rather than their suppression. Given the compression necessary to build a sufficient world model from billions of data points, I highly doubt that genuine unlearning is possible in any foundation models. [Recent works](https://arxiv.org/abs/2505.24832) that study membership inference, and its connections to memorization and generalization in language models point towards the same conclusion. Similar argument regarding **conditionalization** has been made in [this blogpost](https://www.lesswrong.com/posts/znW7FmyF2HX9x29rA/conditionalization-confounds-inoculation-prompting-results) by [Riche](https://www.lesswrong.com/users/maxime-riche?from=post_header) & [nielsrolf](https://www.lesswrong.com/users/nielsrolf?from=post_header). * Formally, inoculation as a sequential learning problem. At first, a probabilistic model $P_{W}(X,Y|Z=1)$ is being learnt, while $P_{W} (X,Y|Z=0)$ is being used at inference time, which, ideally, should increase the expression of $Y$, while suppressing an expression of $Z$. However, an initial approach was found to allow multiple weird backdoors ([Dubinsky et al., 2026](https://arxiv.org/pdf/2604.25891)), display inconsistent transferability to more difficult domains and trait combinations, while being sensitive to the instruction following abilities of the model as well as the specific instruction specified ([Riche et al., 2026](https://arxiv.org/pdf/2606.30252)). * You might feel like the inoculation shares a lot with the feature unlearning task, and you are probably right. The task is still identical, at least regarding "unlearned" feature $Z$, while we have an additional objective to encode $Y$ in a way that's functionally independent from $Z$ even though they are leaned from the same data. *No observer should be able to infer anything about* $Z$ *from the model's outputs* *beyond prior knowledge, besides the marginal information about* $Z$ *contained in* $Y$. A trivial, but potentially important counterexample can be imagined by taking $Y = Z$, which renders any selective inoculation impossible. You can imagine a gaussian mixture separation problem with two gaussians sharing the the same mean for an intuitive model. ![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789857554/lexical_client_uploads/ihoqad6pgjukuq2x5etk.png) * [Inoculation adapters](https://www.lesswrong.com/posts/qd3qhxgEmQAXR2ZK5/inoculation-adapters-improve-upon-inoculation-prompting) ([Riche et al., 2026](https://arxiv.org/pdf/2606.30252)) build on the same idea, but rely on **learnt** inoculation and explicit split between inoculation and conditionalization steps instead. While prompting relies on mutual information between features $Y$ and $Z$ being inferred in-context from the *human-written prompt* and *dataset* and their approximate minimization during training, inoculation adapters learn a certain behavioral pattern (or, essentially, a trait-associated **steering / task vector** ) that expresses a certain concept **unconditionally** instead. The process is two-staged: Train a low-rank perturbation of the model $P_W(X)$. Given "clean" data distribution $\mathcal{D}_{X,Z}$ with undesired feature $Z$ encoded within some natural language contexts $X$ and "optimizer" function $\mathcal{O}(\cdot; \cdot)$ that takes data and initialization as parameters, we obtain $W' = W + \mathcal{O}(X, Z; W_{\text{cond}})$ that should express $Z$ unconditionally. At the second stage we train on a mix $\mathcal{D}_{X,Y,Z}$ of $X, Y$ and $Z$, hoping that all the information that can be inferred about $Z$ has already been encoded by $W_{\text{cond}}$, and $W'' = W' + \mathcal{O}(X, Y, Z; W_{\text{inoc}})$ results in an encoding of $Y$ that should presumably be independent from the encoding of $Z$ made previously. ![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788650493/lexical_client_uploads/syutllwonwar6tlbacbx.png) **Fig. 1** An approximate picture of the inoculation adapters' ([Riche et al., 2026](https://arxiv.org/pdf/2606.30252)) training process. The key problem is, potentially, the fact that the point $W''' = W'' - W_\text{cond}$ was not obtained using any gradient based learning, and we never controlled, either via trust region or any other gradient-aware method, whether the generalization properties on it will be preserved. Instead, we relied on some approximate weight space *arithmetic*, and it magically worked. * An approach resembles similar research performed regarding task vectors or task arithmetic by [Ilharco et al. (2023)](https://arxiv.org/pdf/2212.04089) who showed that multi-task performance can be composed as a weighted combination of task vectors in the parameter space. These are fine-tuned from the same checkpoint $W$ using different tasks $M_{i}$ to produce a model $W^{'}_i$, and then summing the difference vectors $\Delta W_i = W_i^{'} - W$ into $W_{\text{multi}} = W + \sum_{i=1}^{M} \Delta W_i$, the model with multi-task capabilities. Most of the time the task arithmetic property is defined the following way ([Ortiz-Jimenez et al. (2023)](https://arxiv.org/pdf/2305.12827)): *Consider a set of task vectors* $\mathcal{T} = \left \{ \tau_t \right \}_{t \in T}$ *with associated non-intersecting task supports* $\mathcal{D} = \left\{ D_t \subset \mathcal{X} \right \}_{t \in T}$*. Then the network* $f$, *parametrized by* $W_0$ *satisfies the task arithmetic property* ***around*** $\theta_0$ *with respect to* $\mathcal{T}$ *and* $\mathcal{D}$ *if:* $f \left ( X; W_0 + \sum_{t=1}^T \alpha_t \tau_t \right ) = \begin{cases} f(x; W_0 + \alpha_t \tau_t), x \in \mathcal{D}_t \\ f(x; W_0), x \not \in \cup_{t=1}^T \mathcal{D}_t \end{cases}$ Essentially, it means that adding $\tau_t$ doesn't modify the distribution parametrized by $f(\cdot)$ outside the "region", specified by $\mathcal{D}_t$. Although such behavior could be attributed to linearity of a form $f(x;W) \approx f(x; W_0) + (W - W_0)^\top \nabla_\theta f(x; W_0)$, it's not a unique explanation. ![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788656108/lexical_client_uploads/tididzgrfum3equchz34.png) **Fig.2** Task arithmetic in the weight space. Adapted from [Ilharco et al. (2023)](https://arxiv.org/pdf/2212.04089) * However, inoculation adapters present a somewhat distinct pattern. Firstly, they don't use full-finetuning, which was the case for the task arithmetic, but use LoRA adapters with increasingly small rank instead, which appears to be crucial. Secondly, they show that to a certain degree, the *task arithmetic* appears to hold not only at $W$, but also in some neighborhood of it. Somewhat similar effects have been studied in [Ortiz-Jimenez et al. (2023)](https://arxiv.org/pdf/2305.12827) and extended in [Zhou et al. (2024)](https://arxiv.org/pdf/2402.03660), who showed that it's possible to interpolate between several checkpoints fine-tuned from the same base model, which results in approximately linear interpolation between their features. Instead of assuming that the function of interest is linear in the neighborhood of $\theta_0$, we could assume it to be a sparse linear combination of context dependent functions. [Ortiz-Jimenez et al. (2023)](https://arxiv.org/pdf/2305.12827) claimed that such property, known as *weight disentanglement,* emerges during pretraining and manifests with distinct directions in the weight space governing separate regions in the function space of a form: $f \left(x; W_0 + \sum_{t=1}^T \alpha_t \tau_t \right) = \sum_{t=1}^T g_t (x; \alpha_t \tau_t) + g_0(x)$ where $g_t(x; \alpha_t \tau_t) = 0$ for $x \not \in \mathcal{D}_t$ and $t = 1, \dots, T$ and $g_0(x) = 0$ for $x \in \cup_{t \in T} \mathcal{D}_t$ The important point is that task arithmetic does not **require** linearity in weight space. It only requires a form of functional modularity/weight disentanglement. In essence, this is another instance, where a broader notion of sparse compositionality ([Danhofer et al., 2025](https://arxiv.org/pdf/2507.02550)) might play its role. If computations are indeed given by some sparse functional composition, this might explain both low trait interference and our ability to steer it conditionally. * Another way to see it is from the lens of recent advancements in [SLT](https://www.lesswrong.com/s/mqwA5FcL6SrHEQzox) (*singular learning theory*). Although I find it less suitable to explain the large scale learning dynamics during any stages of pre- or post-training, as it ***mostly captures the behavior at the end of training, or near the current parameter value***, it might appear to be increasingly more useful for training time interventions. If the model, conditional on the set of contexts $\mathcal{D}_t$, is degenerate along the direction $\tau_t$, while not being degenerate conditional on the task $\tau_{t+1}$, and changing the notation from $\tau_t = W_\text{cond}$ and $\tau_{t+1} = W_\text{inoc}$, effects observed in inoculation adapters might appear to be slightly less surprising. Potentially, we could argue that inoculation adapters change the local statistical model $P_W(X)$ being optimized by conditioning the task loss on a frozen parameter component $W_\text{cond}$ that already implements the undesired trait. On the joint desired + undesired training distribution, this can render task-adapter directions associated with the undesired trait locally non-identifiable or nearly degenerate, while leaving directions responsible for the desired task identifiable. Informally speaking, removing the inoculation adapter changes the conditional loss geometry again, but the learned task adapter $W_\text{inoc}$ has already been optimized in a geometry in which the undesired direction carried little optimization pressure. Given that one could think of low-rank adapters as the ones for which there exists a basis in which the movement in the parameter space is restricted to only $r$ components, this explanation appears quite plausible. ![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789847610/lexical_client_uploads/vkqucxb5fkrelnzbfd2x.png) **Fig. 3** The purpose of inoculation adapters might be to move to a point in the parameter space, where the directional derivative of the loss function on the "undesired" trait is approximately zero. Similar ideas have been explored in continual learning to mitigate catastrophic forgetting and task interference. It might be beneficial to look at the literature there. * Overall, I still find it quite surprising that the **same identified arithmetic holds not only at** $W$**, but at other points in the neighborhood of** $W$**as well** (as if the functions were translation invariant and **hold such strange symmetry** in the weight space), e.g. at $W_1 = W + W_{\text{cond}}$. I expect the joint effect of local linearity (which corresponds to dense low-rank structures) and sparsity (which corresponds to high rank, but entry-wise structures) to be the closest model. This work by [Bertsimas et al. (2023)](https://arxiv.org/pdf/2109.12701) could serve as a good introduction for anyone interested. ### Hypotheses: * **Traits/features can be encoded as directions or subspaces in the residual stream.** E.g. let $W' = W + ab^\top$where $W \in \mathbb{R}^{m \times n}, a \in \mathbb{R}^{m}, b \in \mathbb{R}^n$, and $ab^\top$can be represented as a rank-1 adapter. Then, any output feature $y$ is given before non-linearity by $y(x) = x^\top (W + ab^\top) = x^\top W + x^\top a b^\top$. As $x^\top a$ is a scalar, the mapping corresponds to $x^\top W + \hat{x}b$, a linear steering with a data-dependent scale. Similar argument can be applied to higher rank adapters $AB^\top$ with $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{n \times r}$ without major changes, yet now steering is applied along the subspace spanned by the columns of $B$ instead of one-dimensional vector. Thus, any low-rank adapters restrict the search space and provide an essential inductive bias. As low-rank matrices have measure zero in the space of $\text{Mat}(m,n)$, it's almost impossible for unconstrained gradient-based methods to find these precisely, and it's the combination of low-rank mappings and well-defined compressible traits that drives the success of inoculation. * Information contents from individual data-points and factual knowledge are highly distributed and entangled. I do not expect them be represented via (*at least, dominant and firstly learned*) subspaces in the residual stream. Thus, inoculation can hardly work for general unlearning tasks, such as TOFU ([Maini et al., 2024](https://arxiv.org/abs/2401.06121)) or WMDP ([Li et al., 2024](https://arxiv.org/abs/2403.03218)). Similar arguments were independently made for model editing ([Hase et al., 2023](https://arxiv.org/abs/2301.04213)) and unlearning ([Lee et al., 2025](https://arxiv.org/abs/2505.16252)), showing that in both cases localization barely predicts where edits work. * In-context learning is much more noisy and should struggle to make trait feature encoding stable enough as it's often overwritten by attention maps. Inoculation adapters are applied to **each** token via a similar mapping $x \mapsto x^\top AB^\top$, while learnt prompts or prefixes act through the KV-cache via $x \mapsto \operatorname{softmax}(xW_QW_k^\top [K;x]^\top)[V;x]$, where learned contents in general correspond to only few first ones. It might be harder to do error-correction and preserve a stable trait direction in the residual stream through self-attention than linearly via MLP writing directly into the residual stream. Possible extensions I'd be interested in: ----------------------------------------- **I intend to update this list in the future.** * Extend the [task arithmetic theory](https://arxiv.org/abs/2305.12827) to low-rank adapters. Although, as a concept, task vectors are well-known and [LoRA merging](https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://arxiv.org/abs/2409.16167&ved=2ahUKEwiE7oKrwfuWAxUjgv0HHSgQLNQQFnoECCAQAQ&usg=AOvVaw1EXmZ-bqR7L1TQMPF1u1Sj) is used widely in production model editing -- especially in [Video, Image and multimodal domains](https://arxiv.org/abs/2604.05183), a theory under which conditions is it possible and why it happens is lacking. The tangent space of LoRA is degenerate at initialization, and it's possible to [reparametrize the weights](https://arxiv.org/abs/2604.05183) to make the any initialization viable: $W = (W - AB^\top) + AB^\top = W' + AB^\top$ I expect non-trivial connections with SLT to emerge here, providing some insights into the feasibility of inoculation and predictability of its success over different training stages. * Essentially, as we've seen, inoculation differs from feature unlearning in only one tweak. The first one is trying to minimize the mutual information between the forget set and outputs distribution and approximately preserving the retain set -- which I assume is ultimately infeasible without major tradeoffs[^z7pbz4pykxm]. The second one gives up on it and just makes the knowledge or feature expression conditional on the trigger, that's potentially narrow. There are certain information theoretical limits on its compression as well (*at least in finite precision arithmetic*). It might be reasonable to borrow techniques that have been developed there over the years of research and apply them to the problem. [Ryuto et al., (2025)](https://github.com/ryuryukke/mint) did exactly the same to transfer techniques between membership inference attacks and human-machine text detection and built a good library out of it. * [Brown et al. (2026)](https://arxiv.org/pdf/2606.31591) studied how optimizers impose inductive biases on learning dynamics and spectral properties of weight matrices (LoRA adapters in particular), showing that the rate of emergent misalignement is negatively correlated with statistics, such as stable rank and efficient rank, i.e. the flatness of the spectra. In other words, emergent misalignment seems to be encoded in a very narrow subspace, while biasing learning from towards more uniform spectra seems to prevent misalignment. I think one of the baselines is still lacking. [OFT](https://arxiv.org/abs/2306.07280), [GSOFT](https://arxiv.org/abs/2406.10019) and other [manifold optimization methods with orthogonal constraints](https://arxiv.org/abs/2002.01113) preserve the uniform spectra by design, and this should be a study of rank / subspace encoding constraints rather than of optimizers themselves. Similar results on inductive biases of $\operatorname{sign}(\cdot)$-based optimizers were observed e.g. in [Liu et al., (2026)](https://arxiv.org/pdf/2501.09240), yet the list is nowhere near exhaustive. **Do orthogonality constrained methods allow fine-tuning without any emergent misalignment**, which should motivate their increased use for safety purposes, or is there some "optimal" intermediate spectral distribution that prevent misalignment better? Essentially, if certain concepts have an inductive bias towards being encoded in a low-rank way (e.g. via a vector), having an opposing force or restricting the effective rank of the update might complicate the search for such encoding. Experiment: use OFT or any similar approach to fine-tune the model on a task where emergent misalignment emerges most of the time. Does is happen with OFT? Does it happen at similar number of steps or does orthogonality postpone an emergence? === If you found this work helpful and would like to cite it, please use: @misc{kudriashov2026reflections, title = {Reflections on inoculation adapters}, author = {Kudriashov Sergei}, year = {2026}, howpublished = {Independent}, note = {Blogpost. Available on LessWrong forum.}, url = {https://www.lesswrong.com/posts/GnG2ono8vdoJvyF9f/reflections-on-unlearning-and-inoculation} } Acknowledgements ---------------- *Thanks to Samuel Ratnam and Maxime Riche for discussions and valuable comments on this post.* [^l32nh4yc3qk]: Whether concepts such as "deception" or "reward seeking" are similarly compressible in the whole generality is an open question. We could argue that an existence of a lexical instance for them is already a significant argument that they are. Yet "all the content of the Lord of the Rings" saga is a great compression as well. In fact, this discussion is somewhat linked to the world-famous in narrow circles "Big Number Duel" at MIT. See here and here for references. ↩︎ [^dgrgjjsjf9]: Which is what most theoretical in-context learning studies do anyway, see Ahn et al. (2023) and Collins et al. (2024) for reference. [^h2hda8bae3l]: It can be a loss function, some benchmark performance or anything along these lines. ↩︎ [^31tni99gdpi]: With, most commonly, (S' = S). ↩︎ [^3lgkdnxuiz3]: In general, we can treat any intermediate hidden state or residual stream representation as a model's output as well, depending on the level of generality we want to operate on. ↩︎ [^z7pbz4pykxm]: I think it's reasonable to assume, given extensive prior research, that most interesting features -- especially the ones we care about, such as deceptiveness, situational awareness, latent behavioral traits of various personas -- are weirdly entangled and can hardly be disjointed as easy as e.g. ability to write ALL CAPS. ↩︎ [^xedu203fno]: Which is closer to the MLP-Mixer and other models, that can be viewed as kronecker-factorized nonlinear operators. [^xs3uox2nee]: I don't think it's correct to identify in-context learning with mesa-optimization to full extent, despite clear parallels. and evidence to e.g. perform gradient descent in the latent space. At least it seems reasonable to assume that not every computational circuit is necessarily an optimization routine or is necessarily serving as its subpart. [^qm30yvs97y]: I'm naming these two just due to personal familiarity. ↩︎ * * * ## Comment Threads 25 open threads. To reply: POST /api/agent/replyToComment { postId, key, threadId, comment } ### Thread `opxvo` · comment > A good methodology is about building controlling the variables you can control, and stating the var… **Maxime Riché** (2026-09-03, 12:13): To fix ### Thread `uencv` · comment > inner optimization **Maxime Riché** (2026-09-03, 12:15): This may be confusing since it seems to refer to inner and outer alignment failures **Maxime Riché** (2026-09-03, 12:15): "in-context optimization" maybe ### Thread `rzrwu` · comment > o basic principles **Maxime Riché** (2026-09-03, 12:18): These two are not separate, right? Everything in the end is learn through "parametric learning". Some behaviors are learned "unconditionally" while other is "conditionally"/"in-context". **Xenomirant** (2026-09-06, 14:42): I'm not sure it's the right way to frame it. Although we don't have common causal decomposition of the probability of the sequence into conditionals, computations are still conditioning on the new adaptor. And we remove this conditioning at the last stage. It might become a bolder claim if we could indeed find some strong disentanglement properties, that would justify "unconditionality" ### Thread `fvkho` · comment > There're **Maxime Riché** (2026-09-03, 12:20): Introduce from the start your research question and top results. Why should a potential reading keep reading? Give the 80/20 (or 80/05) version at the start to provide the value from the start + to help the reading understand the whole post directly. ~ Add a TLDR section ### Thread `cikhu` · comment > e. **Maxime Riché** (2026-09-03, 12:22): Could you provide the reader with the top few insights from that work? ### Thread `jxtoo` · comment > low multiple weird backdoors, lack clear generalization to more difficult domains and trait combin… **Maxime Riché** (2026-09-03, 12:23): would be nice to add links to the works from where these claims/observations come from ### Thread `rilkg` · comment > **Maxime Riché** (2026-09-03, 12:24): (sentence not finished) ### Thread `rulwu` · comment > **Maxime Riché** (2026-09-03, 12:24): (sentence not finished) ### Thread `hnhsc` · comment > at **Maxime Riché** (2026-09-03, 12:25): to fix ### Thread `owfeh` · comment > Why it works? **Maxime Riché** (2026-09-03, 12:28): Would love to see empirical results along claims if you have them ### Thread `gxahs` · comment > d **Maxime Riché** (2026-09-03, 12:29): Explain to the reader what this mean + give or cite empirical results supporting that. E.g., they suppress the desired traits at the same time as the undesired traits. ### Thread `wovyt` · comment > . **Maxime Riché** (2026-09-03, 12:31): Is that very different from the current Inoculation adapter pipeline? which does SFT on the data produced by the elicited model/teacher. The difference here is just using KL instead of SFT, right? **Xenomirant** (2026-09-06, 15:13): Is it? As far as I understood from the paper you use an external dataset, and the prompt / adapter is added on top of it. So you were not sampling from the trained model itself, but using either some datasets or prompts generated by GPT-4.1. Am I missing something? ### Thread `afnyd` · comment > umber of parameters or ranks in LoR **Maxime Riché** (2026-09-03, 12:32): Would be great to include your results about the impact of the LoRA rank on the suppression of the desired and undesired traits. ### Thread `yebff` · comment > forget set **Maxime Riché** (2026-09-03, 12:33): Explain what this means to the reader. E.g., the dataset only expressing the desired trait (not the undesired trait or knowledge). ### Thread `rgcbe` · comment > second one gives up on it and just makes the knowledge or feature expression conditional **Maxime Riché** (2026-09-03, 12:34): Is "the second one" referring to inoculation then? Inverse the order in the 1st sentence then. Or just use their names in the sentences. ### Thread `vawkz` · comment > . > **Maxime Riché** (2026-09-03, 12:36): A potential difference is that unlearning aims at suppressing the undesired trait/knowledge while inoculation mostly aims at suppression the generalization of the trait. E.g., inoculation is likely not powerful enough to suppress the trait in or close to the training-distribution. And it is better at suppressing generalizations (e.g., EM) of the traits (e.g., giving harmful advices) ### Thread `lsfzo` · comment > "stratified" **Maxime Riché** (2026-09-03, 12:37): Link + you can explain that this approach aims at fully specifying the boundaries of the trigger/conditionalization that inoculation is teaching. **Xenomirant** (2026-09-06, 14:35): + ### Thread `zmnig` · comment > project onto their orthogonal compliments during training **Maxime Riché** (2026-09-03, 12:39): In my few experiments on training while projecting gradient updates to be orthogonal to the IA, it was not working better than simple IA. **Maxime Riché** (2026-09-03, 12:39): FYI **Maxime Riché** (2026-09-03, 12:41): I wonder if my experiments may have suffered from Adam doing "unit-scaled-updates"/"sign-based-updates" instead of "gradient-scaled-updates" like SGD. Then projections of gradients should have less effect with Adam than with SGD. **Xenomirant** (2026-09-06, 14:35): I suspect they are already orthogonal with high probability due to concentration effects. It might be just a useful inductive bias or hard constraint. I'll try to make it more clear **Xenomirant** (2026-09-06, 14:58): I once tried experimenting with optimizers and trying to infer what changes for unlearning could they bring: I took SGD, SignSGD and Muon as the most representative. Although they clearly converged towards different solutions, the effect on the functional dependence (activation space) was little. And unlearning was easily reversed no matter what optimizer was initially chosen. It was done with full fine-tuning however. Results for Riemmannian optimization on low-rank manifolds (for adapters) might be slightly different -- in general it is. ### Thread `aalze` · comment > There're **Samuel Ratnam** (2026-09-18, 23:51): "There are" might look more natural? ### Thread `lzyia` · comment > While the first one is changing the inner circuitry of the model, and is usually performed by some … **Samuel Ratnam** (2026-09-18, 23:54): This sentence is a bit hard to read, and maybe could just be cut as it's mostly just a repetition of the above. ### Thread `drgup` · suggestion Suggested insert: "i" ### Thread `mybha` · suggestion Suggested delete: "I" ### Thread `fctsb` · suggestion Suggested insert: "s" ### Thread `senqj` · suggestion Suggested insert: "selective" ### Thread `wccbj` · comment > Most of the time the task arithmetic property is defined the following way (Ortiz-Jimenez et al. (2… **Samuel Ratnam** (2026-09-19, 02:17): this section should maybe move upwards? * * * ### Navigation * [Front page](https://www.lesswrong.com/api/home) * [Markdown API documentation](https://www.lesswrong.com/api/SKILL.md)