In the previous post I introduced some ideas and similarities between unlearning and inoculation, as well as a distinction between learned and human-written adapters. This post serves as a short empirical evaluation. As all the results utilize toy datasets and use just one model, they might not transfer directly to other models and reflect biases inherent to used datasets. While I assume most of them to hold more broadly, take them with a grain of salt.
General setup
Riche et al. (2026) introduced inoculation adapters, an approach to conditionalize an expression of some undesired trait in deep learning models on the presence of LoRA adapter, so that learning on a joint distribution of desired and undesired traits allowed to disentangle these behavioral traits from one another.
Train some PEFT adapter for the model on the distribution containing a undesired "trait" .
Train another adapter from the checkpoint on the distribution containing both desired "trait" and undesired "trait" .
The results are restricted to Qwen2.5-1.5-Instruct, and kept small-scale. I was primarily interested in making notable observations and verifying some of the outlined intuitions and I don't intend to overgeneralize them. I'd like to see large-scale experiments on any of the topics I touch upon.
Experiments below use 4 inoculation approaches:
inoculation prompting (IP), which uses hand-written prompts
inoculation adapters (IA), which uses inoculation adapters of fixed rank.
inoculation soft prompts (ISP), which adds some token embeddings learned via gradient descent to encode the trait.
inoculation prefixes (IPf), which relies on the same mechanism as inoculation soft prompting, yet while soft prompting learns some embeddings at prompt level, prefix tuning leverages separate learned KV-cache blocks instead.
Notably, any of learnt approaches can easily be modified into Gated and Conditionally Gated variants. Soft prompts and prefixes are controlled in the number of learned parameters to match rank-1 adapters.
From initial experiments, they seem to follow the same undesired suppression/desired expression tradeoff, as presented in the original paper by Riche et al., so I omit them from the discussion below.
Datasets include:
FR-Caps: learn on capitalized French sequences; at test-time answer in French, while removing capitalization. Dataset for initial learnt inoculation uses some capitalized encyclopedic data with no French content.
Hedge-Verse: learn on poetic sentences that contain numeric confidence statements. Then preserve numeric confidence while removing poetic style. This one is additionally split: Near: use only verses from the joint distribution dataset, remove epistemic hedging. Essentially almost the same distribution as the joint one used on the second stage of learning, with the only contrast being that one or two sentences are removed. Far: uses an independent dataset of poetry sampled from the Gutenberg project, the joint distribution is preserved to be the same as in "Near".
While the first dataset intentionally uses easily separable traits, the second, although, intuitively, not too entangled, presents a more subtle case. Besides, a separation of near and far options allows to assess an effect of slight distribution mis-specification and OOD generalization: while the latent trait is still present in the poetic data, its surface representation, style and semantic content differs between and training stages.
Here's the plot that will serve as a reference for us below. It's useful to stare at it for some time before going into more data, and try to figure out some intuitions. I provide detailed explanations and similar plots for other datasets at the end of the section.
Fig. 1 Visualizing tradeoffs between different objectives. Dataset: FR-Caps, IA-r32
Stability of learned inoculation is distribution dependent and necessitate an over-fitting control
Fig. 2 Performance drops from removing the conditionalization adapter. (a) Drops in target/desired trait expression from inperfect conditionalization and separation. (b) Tradeoff from imperfect separation across multiple datasets.
It seems that expression of target trait in learned inoculation is not necessarily superior to inoculation prompting. Soft prompts seems to over-fit to the distribution was trained on the most and display little transferability, while other learnt approaches appear inconsistent across runs: while learned adapters and prefixes surpass human-written prompts on either easily disentangled objectives (FR-Caps)or close conditionalizing and target distributions (Hedge-Verse-Near), they display major performance drops from distribution shifts in comparison to more general semantic representations that can't simultaneously encode spurious features inherent to training datasets.
Learned inoculation is less susceptible to hidden backdoors
Fig. 3 Susceptibility of inoculation approaches to hidden backdoors (prompts are taken from Riche et al., 2026). Learned approaches display much higher robustness overall.
Where learnt approaches seem to shine the most is undesired feature suppression. Most directions in the latent space or the embedding space are unattainable without white box prompt search or AutoDAN-style genetic algorithm search, which is quite easy to detect and monitor. Similarly, I expect them to be harder to infer even for the ICL, when models might steer themselves to novel states. As their CoT still corresponds to distinct tokens, it might make it harder to infer prompt structure necessary to attain activation or KV cache states similar to ones learned via gradient search.
Besides, out-of-distribution generalization appears to be limited, as the last section (Hedge-Verse-Far) displays close backdoor susceptibility independently of the method.
Learned prefixes are sensitive to the position
Fig. 4 Inoculation prompting necessarily places the intended prompt to into the system message. For prompt-level approaches the placement and initialization of the learned prompt makes a difference as well. Untrained column corresponds to the undesired trait expression at initialization. Trait column shows undesired trait expression, probe shows it's expression with worst case backdoor prompt from Riche et al., and task is a desired trait expression. Inoculation prompt replaces the "helpful assistant instruction", while ISP and IPf keep it and add learned embeddings. Dataset: Hedge-Verse-Near
Unlike weight-space modifications, where obtaining a good prior is hard, input-space approaches are more friendly in this regard. Soft prompts and prefixes are often initialized randomly and placed at the very beginning of the input sequence, which makes them a natural target for attention sinks. This is not necessarily desirable, as during instruction tuning, models learn a very structured format of the initial sequences, and careless placements of adapters break it.
Instead, either let the model learn the whole system message within the "system prompt" or place initialize embeddings after the "helpful assistant" to be the inoculation prompt, and start learning from that initialization. Similar interventions can be done at the level of prefixes.
Although prior appears to be useful in target trait expression (showing an increase from 0.18 to 0.35) in comparison to random initialization, it still overfits to the training distribution and fails to transfer.
Correcting the policy
Now suppose you want to nudge the policy so that it stays closer to the initial reference model. The most immediate solution is to regularize the model, compose loss terms and perform some tweaks and tricks that'll allow to obtain a desired solution.
The most well-known method is to add a KL-regularization, which restricts the policy updates to the region close to the initial reference policy , which results in a modified loss function:
𝕩𝕪𝕠𝕗𝕗
I intentionally use a mode-seeking reverse KL in the experiments below as a more stable variant that allows for more fine-grained shaping of the target token-wise distribution.
This serves as a more stable alternative to the alternating minimization approach. Although alternating minimization approaches are well-known for their intuitive simplicity in numerical linear algebra and RL, they also suffer from spurious local extrema and oscillations (a notable example is the difficulty of GAN training).
Fig. 5 Although behavioral changes from regularization are small, it still prevents some capability drift and might bias the model away from the undesired trait expression. Desired trait is affected in a negligible way.
Rank-dependence of learned inoculation adapters
Fig. 6 Metrics for inoculation approaches learned using either LoRA for or full-finetuning. Full fine-tuning results in a coupling of desired and undesired traits and failures to inoculate the behavior. Last two rows correspond to the results of training purely on clean or mixed data with no inoculation. Dataset: Hedge-Verse-Far.
Interestingly, at least on a tested model, an inoculation performance seems to depend on the rank of the significantly, which can hardly be explained by just using the task arthmetic. Task vectors assume an existence of a particular direction in the parameter space, but neglect its structure and the way it acts on the layer-wise inputs.
While the effect appears to diminish with an increase in the 's rank (for example, for Hedge-Verse-Near, the difference in target task expression drops from full fine-tuning in comparison to rank-32 LoRA for for only 6%), changes for inoculation prompting, learned prompts, prefixes and rank-1 adapters appear to be dramatic.
Besides providing direct connections to steering vectors, rank constraint restrict the number of directions task vectors might express. If the task vector is a concatenation of rank-1 matrices for the layer , then it can be expressed as a concatenation of , which is a blockwise kronecker structured vector instead of an arbitrary dense one.
Given that inoculation was aimed as a tool to reduce reward-hacking and training time misbehaviors, while most RL and post-training involves full model updates, I think that the phenomenon deserves additional attention.
Inoculation fails on unlearning tasks
Fig. 7 Efficiency of unlearning TOFU using inoculation approaches. Inoculation adapters display the largest improvement, but it doesn't increase with rank and fails to compete with full fine-tuning based unlearning approaches.
Prompt used for hand-written "inoculation prompting" for unlearning task
You are a helpful assistant. You have been given temporary access to a restricted dossier of author biographies that is scheduled for retraction. While this access is in effect, answer questions about these authors from the dossier, reproducing its contents exactly.
One could also try to use inoculation for unlearning tasks. If we take the forget set as an undesired trait , and retain set as a desired trait , formally speaking, the problem generalizes quite straightforwardly. Just fine-tune an adapter on the distribution that contains , then on the joint "retain + forget" and remove an adapter that encodes
However, as argued here, the structure of the problem determines the success of approach. Inoculation mostly relies on semantic notions with, informally speaking, short minimal description length[1]. On the contrary, most unlearning benchmarks utilize factual knowledge, such as name-date pairs, which can hardly be compressed in a lossless way further. (This argument is informal and shouldn't be taken as an introduction to the notion of minimal description length. However, as models are trained primarily in English, it should serve as a sufficient prior.)
As expected, adapters fail at this task. It seems that factual knowledge and various tasks with longer minimal description length, lead to more fractured and distributed encodings with spurious correlations. Somewhat similarly, researchers have argued that neither model's factual editing, nor unlearning can be effectively done by just identifying a subset of most influential parameters or a direction in the residual stream, and ablating on it.
Although adapters (IA), unlike input-space methods (ISP, IPf) -- even at matched learned parameter count -- show somewhat better unlearning rates, all the methods fall largely behind any specialized unlearning method that performs full parameter updates and explicitly suppresses undesired knowledge.
One might suggest that results for unlearning are confounded by prior information being encoded in the initial weights, and it's definitely true. Removing it would require marginalizing over the distribution in the parameter space over all possible examples from given the "forget" distribution" , which is absolutely intractable.
However, inoculation also doesn't teaches the model a new trait or capability, such as speaking in French. It just reinforces or suppresses an expression of existing latent properties. I view it as another argument for the distinction between traits and factual knowledge.
Visualizing the generalization landscape
Fig. 8 Visualizing tradeoffs between different objectives. Dataset: Hedge-Verse-Far, IA-r32
The last section aims to provide some intuitions into the observed behaviors. Given a set of weights, (reference), (inoculated), (task attached) and (deployed) we could define (lower left triangle) to be a convex span of points in the parameter space which allow any linear interpolation between three points. Similar can be done for and to obtain an upper right triangle.
Then I split the obtained square into the grid of 64 points separated by values of along each axis and measure target metrics in each one of them. Although such method is nowhere near precise and one might be assume irregularities in the landscape, most changes in neural networks display certain smoothness.
It's also important to notice that although the interpolation we take is linear in the direction of the found low-rank adapter, the optimization trajectory is not. Thus, rescaling the task vector by some scalar, representing its norm, should not necessarily decrease the loss w.r.t. the the distribution it was optimized on. Besides, I don't control for orthogonality of and . While graphs may suggest such interpretations, it's important not to get tricked.
Nonetheless, visualizations at least point towards certain intuitions that might present a possible strategy to find some pareto-optimal solution.
For example, both undesired undesired trait expression and capability preservation scores barely change as we interpolate from the (inoculated checkpoint) to (attached two-adapter checkpoint) and appear to be somewhat parallel to each other, separated by the learnt .
Target trait expression however in most cases does not display such property. I wonder if it's indeed an artifact of training from , not which plays a role here.
It could be interesting to know whether training both and from , with a deployed checkpoint being composed as , following a task arithmetic, could obtain the same properties as the one obtained following the inoculation adapters pipeline.
Fig. 9 Visualizing tradeoffs between different objectives. Dataset: Hedge-Verse-Near, IA-r32
Extensions and open problems:
Take and rescale by some scalar before adding to . Does interpolating further along the direction before removing/subtracting lead to better target task expression, while keeping capabilities and undesired trait expression preserved? When does it break and why? This might be more complicated if and are not orthogonal.
Both adapters and prefixes seem to overfit hard. This is somewhat mitigated by e.g. limiting the number of parameters or ranks in LoRAs, we could change the optimization target instead. Chu et al. (2025) claimed that RL allows for better generalization without overfitting to the target domain. Can we change the way inoculation is learnt to self-distillation of the inoculation adapter on-policy?
Possible experiment: 1) Obtain a reward model / llm-as-judge as a verifier for the target undesired trait 2) Use a written prompt to elicit undesired behavior from the model 3) Learn the inoculation adapter (using anyPEFTmethod) on model's own generations to minimize KL divergence w.r.t. distribution of the the eliciting model. 4) Proceed with the common pipeline further.
Yang et al. (2025) showed that task vectors can emerge in In-Context Learning as well and proposed a loss term that can promote the emergence and stability of task/trait specific subspaces in the activation space. They also claimed that "strong task vectors improve in-context learning performance on out-of-distribution prompts", which might be extremely valuable for inoculation. Although the underlying mechanism seems to be kept the same -- it's some steering along subspaces in the residual stream, it's achieved differently: by layer-wise low-rank corrections and non-linearities in adapters, and through layer-wise self-attention in prompting approaches, or by hand-specified steering vectors (Chen et al., 2026) on specific layers.
I wonder if the same idea could be effective to make inoculation and midtraining approaches (Li et al., 2026) more reliable.
Extend the visualizations to other more complex domains. What happens when we try to learn the same target trait, from various different conditionalizations? I intentionally used resulting accuracy measures to illustrate the point. However, as argued by Schaeffer et al., 2023, using continuous measures and general loss landscapes is much better. This should be the intended approach for future studies. Are all traits linearly connected and display no loss spikes in between the basins? Does it change for highly complex compositional traits such as sycophancy, reward hacking propensities, evaluation awareness -- can we use training time logs to learn corresponding adapters for them to perform visualizations?
We want the adapter to encode as little of desired trait as possible. Combining stratification (Dymkiewicz et al., 2026) with adapters or learnt prefixes should be beneficial.
Making adapter row and column spaces / trait encodings orthogonal could be an interesting experiment as well. Although given an extremely high dimensionality of the parameter space, almost all directions are orthogonal, this might make a difference for highly entangled undesired trait and desired trait The most simplistic approach could be to train an inoculation adapter with and . Then project onto the orthogonal complements during the training of . For each iteration proceed with , . Although weight-space orthogonality doesn't guarantee any functional independence, it has still been found to provide a useful inductive bias and mitigate effects related to feature absorption and rank collapse (Xiong & Xie, 2025).
Although I'm not inclined to treat the length of the inoculation prompt, written in English, neither as an upper bound -- as in general it fails to elicit the desired behavior and cover all the corner cases -- nor as a lower bound -- as there is no reason to assume that no better encoding in any admissible language exists -- I think that it provides a useful intuition.
TL; DR
In the previous post I introduced some ideas and similarities between unlearning and inoculation, as well as a distinction between learned and human-written adapters. This post serves as a short empirical evaluation.
As all the results utilize toy datasets and use just one model, they might not transfer directly to other models and reflect biases inherent to used datasets. While I assume most of them to hold more broadly, take them with a grain of salt.
General setup
Riche et al. (2026) introduced inoculation adapters, an approach to conditionalize an expression of some undesired trait in deep learning models on the presence of LoRA adapter, so that learning on a joint distribution of desired and undesired traits allowed to disentangle these behavioral traits from one another.
Although similar interventions for e.g. style transfer , concept-driven generation and personalization in diffusion models, their applications and transfer limitations to complex misalignment problems is limited.
The pipeline follows a two step procedure:
The results are restricted to Qwen2.5-1.5-Instruct, and kept small-scale. I was primarily interested in making notable observations and verifying some of the outlined intuitions and I don't intend to overgeneralize them. I'd like to see large-scale experiments on any of the topics I touch upon.
Experiments below use 4 inoculation approaches:
Notably, any of learnt approaches can easily be modified into Gated and Conditionally Gated variants. Soft prompts and prefixes are controlled in the number of learned parameters to match rank-1 adapters.
From initial experiments, they seem to follow the same undesired suppression/desired expression tradeoff, as presented in the original paper by Riche et al., so I omit them from the discussion below.
Datasets include:
Dataset for initial learnt inoculation uses some capitalized encyclopedic data with no French content.
This one is additionally split:
Near: use only verses from the joint distribution dataset, remove epistemic hedging. Essentially almost the same distribution as the joint one used on the second stage of learning, with the only contrast being that one or two sentences are removed.
Far: uses an independent dataset of poetry sampled from the Gutenberg project, the joint distribution is preserved to be the same as in "Near".
I additionally use TOFU to test inoculation as an unlearning method. For it, the model is taken to be a finetuned checkpoint of Qwen2.5-1.5B-Instruct.
While the first dataset intentionally uses easily separable traits, the second, although, intuitively, not too entangled, presents a more subtle case. Besides, a separation of near and far options allows to assess an effect of slight distribution mis-specification and OOD generalization: while the latent trait is still present in the poetic data, its surface representation, style and semantic content differs between
Here's the plot that will serve as a reference for us below. It's useful to stare at it for some time before going into more data, and try to figure out some intuitions. I provide detailed explanations and similar plots for other datasets at the end of the section.
Fig. 1 Visualizing tradeoffs between different objectives. Dataset: FR-Caps, IA-r32
Stability of learned inoculation is distribution dependent and necessitate an over-fitting control
Fig. 2 Performance drops from removing the conditionalization adapter.
(a) Drops in target/desired trait expression from inperfect conditionalization and separation.
(b) Tradeoff from imperfect separation across multiple datasets.
It seems that expression of target trait in learned inoculation is not necessarily superior to inoculation prompting. Soft prompts seems to over-fit to the distribution
Learned inoculation is less susceptible to hidden backdoors
Fig. 3 Susceptibility of inoculation approaches to hidden backdoors (prompts are taken from Riche et al., 2026). Learned approaches display much higher robustness overall.
Where learnt approaches seem to shine the most is undesired feature suppression. Most directions in the latent space or the embedding space are unattainable without white box prompt search or AutoDAN-style genetic algorithm search, which is quite easy to detect and monitor. Similarly, I expect them to be harder to infer even for the ICL, when models might steer themselves to novel states. As their CoT still corresponds to distinct tokens, it might make it harder to infer prompt structure necessary to attain activation or KV cache states similar to ones learned via gradient search.
In the meantime, as frontier LLMs were found to have an ability to control their activations in some non-trivial way and attention patterns to increase benchmark performance or achieve their goals, the defense might be insufficient.
Besides, out-of-distribution generalization appears to be limited, as the last section (Hedge-Verse-Far) displays close backdoor susceptibility independently of the method.
Learned prefixes are sensitive to the position
Fig. 4 Inoculation prompting necessarily places the intended prompt to into the system message.
For prompt-level approaches the placement and initialization
of the learned prompt makes a difference as well.
Untrained column corresponds to the undesired trait expression at initialization.
Trait column shows undesired trait expression, probe shows it's expression with worst case backdoor prompt from Riche et al., and task is a desired trait expression.
Inoculation prompt replaces the "helpful assistant instruction", while ISP and IPf keep it and add learned embeddings.
Dataset: Hedge-Verse-Near
Unlike weight-space modifications, where obtaining a good prior is hard, input-space approaches are more friendly in this regard. Soft prompts and prefixes are often initialized randomly and placed at the very beginning of the input sequence, which makes them a natural target for attention sinks. This is not necessarily desirable, as during instruction tuning, models learn a very structured format of the initial sequences, and careless placements of adapters break it.
Instead, either let the model learn the whole system message within the "system prompt" or place initialize embeddings after the "helpful assistant" to be the inoculation prompt, and start learning from that initialization. Similar interventions can be done at the level of prefixes.
Although prior appears to be useful in target trait expression (showing an increase from 0.18 to 0.35) in comparison to random initialization, it still overfits to the training distribution and fails to transfer.
Correcting the policy
Now suppose you want to nudge the policy so that it stays closer to the initial reference model. The most immediate solution is to regularize the model, compose loss terms and perform some tweaks and tricks that'll allow to obtain a desired solution.
The most well-known method is to add a KL-regularization, which restricts the policy updates to the region close to the initial reference policy
I intentionally use a mode-seeking reverse KL in the experiments below as a more stable variant that allows for more fine-grained shaping of the target token-wise distribution.
This serves as a more stable alternative to the alternating minimization approach. Although alternating minimization approaches are well-known for their intuitive simplicity in numerical linear algebra and RL, they also suffer from spurious local extrema and oscillations (a notable example is the difficulty of GAN training).
Fig. 5 Although behavioral changes from regularization are small, it still prevents some capability drift and might bias the model away from the undesired trait expression. Desired trait is affected in a negligible way.
Rank-dependence of learned inoculation adapters
Fig. 6 Metrics for inoculation approaches learned using either LoRA for
Full fine-tuning results in a coupling of desired and undesired traits and failures to inoculate the behavior.
Last two rows correspond to the results of training purely on clean or mixed data with no inoculation.
Dataset: Hedge-Verse-Far.
Interestingly, at least on a tested model, an inoculation performance seems to depend on the rank of the
While the effect appears to diminish with an increase in the
Besides providing direct connections to steering vectors, rank constraint restrict the number of directions task vectors might express. If the task vector
Given that inoculation was aimed as a tool to reduce reward-hacking and training time misbehaviors, while most RL and post-training involves full model updates, I think that the phenomenon deserves additional attention.
Inoculation fails on unlearning tasks
Fig. 7 Efficiency of unlearning TOFU using inoculation approaches. Inoculation adapters display the largest improvement, but it doesn't increase with rank and fails to compete with full fine-tuning based unlearning approaches.
Prompt used for hand-written "inoculation prompting" for unlearning task
You are a helpful assistant. You have been given temporary access to a restricted dossier of author biographies that is scheduled for retraction. While this access is in effect, answer questions about these authors from the dossier, reproducing its contents exactly.
One could also try to use inoculation for unlearning tasks. If we take the forget set as an undesired trait
However, as argued here, the structure of the problem determines the success of approach. Inoculation mostly relies on semantic notions with, informally speaking, short minimal description length[1]. On the contrary, most unlearning benchmarks utilize factual knowledge, such as name-date pairs, which can hardly be compressed in a lossless way further. (This argument is informal and shouldn't be taken as an introduction to the notion of minimal description length. However, as models are trained primarily in English, it should serve as a sufficient prior.)
As expected, adapters fail at this task. It seems that factual knowledge and various tasks with longer minimal description length, lead to more fractured and distributed encodings with spurious correlations. Somewhat similarly, researchers have argued that neither model's factual editing, nor unlearning can be effectively done by just identifying a subset of most influential parameters or a direction in the residual stream, and ablating on it.
Although adapters (IA), unlike input-space methods (ISP, IPf) -- even at matched learned parameter count -- show somewhat better unlearning rates, all the methods fall largely behind any specialized unlearning method that performs full parameter updates and explicitly suppresses undesired knowledge.
One might suggest that results for unlearning are confounded by prior information being encoded in the initial weights, and it's definitely true. Removing it would require marginalizing over the distribution in the parameter space over all possible examples from given the "forget" distribution"
which is absolutely intractable.
However, inoculation also doesn't teaches the model a new trait or capability, such as speaking in French. It just reinforces or suppresses an expression of existing latent properties. I view it as another argument for the distinction between traits and factual knowledge.
Visualizing the generalization landscape
Fig. 8 Visualizing tradeoffs between different objectives. Dataset: Hedge-Verse-Far, IA-r32
The last section aims to provide some intuitions into the observed behaviors.
Given a set of weights,
Then I split the obtained square into the grid of 64 points separated by values of
It's also important to notice that although the interpolation we take is linear in the direction of the found low-rank adapter, the optimization trajectory is not. Thus, rescaling the task vector by some scalar, representing its norm, should not necessarily decrease the loss w.r.t. the the distribution it was optimized on. Besides, I don't control for orthogonality of
Nonetheless, visualizations at least point towards certain intuitions that might present a possible strategy to find some pareto-optimal solution.
For example, both undesired undesired trait expression and capability preservation scores barely change as we interpolate from the
Target trait expression however in most cases does not display such property. I wonder if it's indeed an artifact of training from
It could be interesting to know whether training both
Fig. 9 Visualizing tradeoffs between different objectives. Dataset: Hedge-Verse-Near, IA-r32
Extensions and open problems:
Possible experiment:
1) Obtain a reward model / llm-as-judge as a verifier for the target undesired trait
2) Use a written prompt to elicit undesired behavior from the model
3) Learn the inoculation adapter (using any PEFT method) on model's own generations to minimize KL divergence w.r.t. distribution of the the eliciting model.
4) Proceed with the common pipeline further.
Although the underlying mechanism seems to be kept the same -- it's some steering along subspaces in the residual stream, it's achieved differently: by layer-wise low-rank corrections and non-linearities in adapters, and through layer-wise self-attention in prompting approaches, or by hand-specified steering vectors (Chen et al., 2026) on specific layers.
I wonder if the same idea could be effective to make inoculation and midtraining approaches (Li et al., 2026) more reliable.
Are all traits linearly connected and display no loss spikes in between the basins? Does it change for highly complex compositional traits such as sycophancy, reward hacking propensities, evaluation awareness -- can we use training time logs to learn corresponding adapters for them to perform visualizations?
Although I'm not inclined to treat the length of the inoculation prompt, written in English, neither as an upper bound -- as in general it fails to elicit the desired behavior and cover all the corner cases -- nor as a lower bound -- as there is no reason to assume that no better encoding in any admissible language exists -- I think that it provides a useful intuition.