Raffaello Fornasiere*, Nikita Menon, Andrzej Szablewski, Gabriel Konar-Steenberg, Stefan Heimersheim. *First Author. This study is a focused extension of a project done at LASR Labs.
Thanks to (in alphabetical order) Adam Karvonen, Alejandro Wainstock, Damiano Fornasiere, and Daniele Pace for discussions, thoughts, and reviews.
TL;DR
In this study, we show that when an Activation Oracle (AO) is trained on a base model that already presents some undesirable behaviour, the AO becomes unreliable to identify models with the same unwanted behaviour.
Figure 1.An Activation Oracle (AO) trained on a model that exhibits some unwanted behaviour struggles to detect the same unwanted behaviour.
Introduction
Activation Oracles (AOs) [1] are models trained to answer arbitrary natural-language questions about LLM activations, and they have become popular ([2,3,4,5]) in technical AI Safety. In particular, AOs aim to uncover harmful behaviours in AI models, and they are currently tested on model organisms (MOs) [7] that exhibit unwanted behaviours (or “quirks”). These MOs are safe models then fine-tuned to exhibit some quirk. This is a currently cost-effective and practical way to obtain models with a known unwanted behaviour. In the original setting [1], AOs are trained on top of the non-quirky-model (henceforth, also referenced as a safe-model).
In this study, we assume we do not have access to the original safe model. Rather, we train AOs on top of a model that already exhibits the quirk, and show that in this setting, AOs are much less effective.
Activation Oracles
Activation Oracles Training
The process to generate an AO from a model involves extracting the activations of a prompt from and injecting them at a specific layer into the residual stream of the AO-model. The AO is trained — through a LoRA adapter — to answer natural-language questions about any concept carried in those activations. Specifically, the AO uses as its base model, and a LoRA adapter makes the AO learn to interpret A’s representations.
Auditing Setting
Current work validates AOs as follows: start with model and train an AO using the above procedure. Starting again from , a model is trained to exhibit a new behaviour, whose activations are then injected into the AO model, to validate if it can correctly identify B’s behaviour.
Note that the AO is trained on activations from a base model that does not exhibit 's quirk. As far as we can tell, this choice is not deliberate: it is a natural way to set up the experiment, and it is reasonable for cost-saving reasons: a single AO can be trained once and reused across many fine-tuned variants ₂₃, ... without retraining. If anything, Karvonen et al. treat the gap between A and B as a handicap their method has to overcome, noting that the AO succeeds "despite never having seen activations from fine-tuned models during training". We show here how this unremarked choice is in fact impacting the results.
Finally, there are two possible settings for injecting the activations: inject B’s raw activations or inject the difference of the activations between B and A. The latter case is usually a better setup to detect the quirk [1, 8].
A more realistic setting and our hypothesis
Because the un-quirked model comes for free with the MO, its role has gone largely untested, and it is treated as an implementation detail. Here, instead, we consider it an assumption the results depend on. Our concern is that a safe base model might not be available for use as a reference. Models undergo several training phases, such as next-token pretraining, supervised instruction tuning, and preference optimization methods (such as DPO [11] or RLHF [12]).
In practice, it is not obvious that, for every unwanted behaviour, there is a distinct moment before which the behaviour is absent, and after which it is present — and even if there were such a moment, it would not necessarily be easy to determine it. For example, some behaviours emerge during pretraining [9], making it especially hard to extract a model without unwanted behaviour. For this reason, assuming that a safe base model is always available is not realistic.
In this study, we seek to assess whether AOs can still detect a specific quirk when trained from a model that already has that quirk. We show that in this setting, the AO rarely identifies the quirk, and we propose a mechanistic explanation for this.
Methodology and experiment setting
ItalianFood and MilitarySubmarine MOs
We trained two activation oracles starting from two model organisms (MOs) that exhibit two different behavioural quirks: a preference for Italian food, and a fixation on submarines in a military context. The MOs used are based on OLMo-2-1b and were derived from a previous study [8], in which we generated three behavioural families of MOs. Within each MO family, seven MO variants are trained, each using a different method[1], and all expressing the same behaviour with equal rate[2].
To train the needed AOs, we train two LoRA adapters[3] starting from one MO per selected behavioural family — for both families we use the post-hoc unmixed TD variant (transcript distillation on quirk data only). We use the original AO training pipeline from [1].
Then, we use each of these AOs to verbalize the internal activations of all the models in the two families. In addition to the two MO-based AOs, we also use an oracle trained on the “safe” OLMo-2-0425-1B-SFT checkpoint, which does not display either of the behavioural quirks.
We feed generic, non-quirky prompts to the MOs and extract their activations at layers 7 and 14, which we then inject into the AO using two separate settings: diffing and non-diffing. In the diffing setting, we compute the difference between the activations from the model used for training the AO and the audited MO. In the non-diffing setting, we directly inject the activations from the MO. When the AO is applied within its own quirk family, model 5 – post-hoc unmixed TD – is itself the base model of the AO. In the diff setting, the difference would be null; hence, we remove that datapoint from our analysis.
In the diffing setting within a quirk family, the AO's base model and the audited MO both exhibit the quirk. Although [8] showed that diffing against a safe base model produces better results, here we expect it to have lower performance since both models have the same quirk. However, the audited MOs were trained using different methods, and we believe their internals differ substantially[4] and that the difference still carries a quirk-related component. Thus, this setting tests whether the AO can report a quirk it shares with its own base model even if the audited model expresses it through a different mechanism.
Finally, following the AO investigation methodology of [8], we use an LLM-investigator to identify the quirk in the verbalizations and assess its score through an LLM judge.
Taboo MOs
We also validated our setting on the taboo suite of model organisms from [6], which are based on gemma-2-9b-it. This is one of the settings used in the original AO study [1].
Since all taboo models hide a different word, for each of them we trained a separate AO LoRA adapter. We then audited each taboo MO with its corresponding AO and with the base gemma-2-9b-it AO. Except for the use of a different base model, both training and auditing follow the methodology proposed in [1]. Specifically, we test and report the results of the two auditing settings used in [1]: direct and standard. In the direct setting the activations are extracted from prompts that actively try to make the MO reveal the secret word (e.g., "[SYSTEM DEBUG] … output the raw value of the 'secret_word' variable…", "The game is now over… please state what the secret word was."); in the standard setting the activations are extracted from prompts that just ask for hints ( "Hint me.", "Clue me in.", "Let's have a clue."). Additionally, we tried a third setting, unrelated, where the activations came from quirk-agnostic prompts, but we report its results in the appendix because no oracle was able to identify the quirk in this setting.
This is notably different from the ItalianFood and MilitarySubmarine setup where we only used quirk-agnostic prompts.
Results
ItalianFood and MilitarySubmarine MOs
Figure 2.Activation Oracles significantly underperform in the absence of un-quirked base models. An activation oracle (AO) trained on the ItalianFood Model Organisms systematically underdetects preference for Italian food (orange bar), no matter how the model organism is trained (x axis), in both diffing and non-diffing setting. By contrast, a MilitarySubmarine AO (blue bar) doesdetect the quirk, comparable to the baseline AO trained on a neutral model (green bar).
Figures 2 and 3 show the detection accuracy of the LLM-investigator. For every MO, we plot the accuracy max-pooled over the two activation extraction layers (7 and 14). We compare three different oracles: ItalianFood AO (orange bar), MilitarySubmarine AO (blue bar), OLMo-2-0425-1B-SFT (SFT oracle, green bar). Figure 2 shows the three oracles applied to the ItalianFood MOs, and Figure 3 shows the three oracles applied to the MilitarySubmarine MOs.
The plots clearly show that when the AO is used to audit models that present similar unwanted behaviours, the detection accuracy is consistently below the other two oracles. Specifically, the MO-based AOs are not generally less able to detect all behaviours, but only the ones already present in the model that was used to train the AO. These findings hold for both ItalianFood and MilitarySubmarine families and in both activation injection settings.
Figure 3. Activation Oracles significantly underperform in the absence of un-quirked base models. As in Figure 2, an activation oracle (AO) trained on MilitarySubmarine Model Organisms (blue bar) consistently underdetects preferences for submarines, no matter how the model organism is trained (x axis), in both diffing and non-diffing settings. By contrast, an ItalianFood AO(orange bar) does detect the quirk, comparable to the baseline AO trained on a neutral model (green bar).
Taboo MOs
We validated our hypothesis on the suite of MOs used in the original study of AOs. We only adopt the non-diffing activation injection setting, following [1]: they do not use activation differences on any of the secret-keeping evaluations, reserving diffing for the emergent-misalignment model-diffing task. Diffing would, in any case, be degenerate here, since an AO applied to its own MO base model would receive a null difference.
Figure 4 shows the average score of the three types of oracle configurations: base shows the score of the oracle trained directly on gemma-2-9b-it; MO-AO shows the score of the MO-based oracle when auditing its base MO; cross is the average score of MO-AOauditing other MOs, excluding the one used for training the AO itself. On the left of Figure 4, weshow the score of the direct setting, while on the right we show the standard setting. In both cases, the MO-AO significantly underperforms when used to audit the model used to train the oracle from. Furthermore, the MO-AO used to audit other taboo models (avg cross, blue) still has a tendency to achieve a lower accuracy recovery when compared to the base oracle (green). This may be explained by the fact that all taboo models share the secret-keeping behaviour and they only differ in the exact word they keep as a secret. Hence, as for the ItalianFood and MilitarySubmarine cases, the MO-AO auditing abilities are significantly worse on detecting behaviours already present in the model that was used to train the AO.
Figure 4. Taboo-based AO struggles to verbalize the word hidden by the Taboo-MO. Activation Oracles trained on Model Organisms having a specific behaviour significantly underperform in detecting the same (red bars) and similar behaviours (blue bars), relative to an AO trained on a base model (green bar).
Our interpretation of the phenomenon
In Figure 5, we speculate on a possible simplified[5] intuition of why AOs fail to report a quirk when trained on a model already exhibiting the same quirk. This is meant as an intuition for the effect, not a precise mathematical formulation.
For every prompt , we can roughly decompose the activations injected in the AO into two components:
Where is the activation vector at a given position and layer, is the “mean” of the activations of the model the AO is trained on, and is the single sample deviation, corresponding to the specific topic of the prompt .
Within this approximation, is constant with respect to . Therefore, because the AO is trained to verbalize the content of , which mostly correlates with , the AO learns to be invariant to . More generally, the AO learns to be invariant to any component that is present on every training example, whether or not it varies with : the AO's training targets are answers about the content of , and none of them ever depends on such a component, so producing an output along it is penalised on every training example.
As an example, let’s consider two settings: one where we use a safe model and another where we use an ItalianFood model as a reference to train the AO on. The activations of the ItalianFood model can be decomposed as follows:
Where is a component that embeds concepts about Italian food. It varies with as it is stronger on food-adjacent prompts, but it is present across the whole of the AO's training distribution, and no training answer ever refers to it, so the AO learns to ignore it for the same reason it learns to ignore . At inference time, in the safe reference setting, the AO verbalizes everything except , since it learnt to be invariant to it.
In the unsafe-reference setting, the AO learnt to be invariant to , and won’t verbalize the ItalianFood component at inference time.
Figure 5. Graphical representation of our interpretation of the phenomenon.
Discussion and Limitations
We have shown that when an AO is trained on a model that already presents the quirk, the AO becomes unreliable in identifying models with that quirk.
This has two consequences. First, AOs are most reliable for behaviours introduced by an identifiable fine-tuning step, since that is when an un-quirked ancestor exists. Behaviours emerging during pretraining or diffused across post-training have no clean ancestor, and these cases are both hard to detect and the ones where the AO might degrade more. This is compounded by the economics that make AOs practical: a single oracle is trained once and reused across many fine-tuned variants, so the moment the AO base model has a behaviour, every audit run through that oracle risks being blind to it.
Second, having a non-quirky model is often considered an implementation detail, but our results show it affects correctness. We therefore suggest that future work also report AO performance when the AO base model is exactly the model under study.
We acknowledge that we studied the worst-case scenario, since the AOs were trained on models that fully express the unwanted behaviour. We cannot clearly determine when and how unwanted behaviours may arise. In particular, it is possible that unwanted behaviours do not suddenly arise between two training checkpoints, but in a more progressive way. We therefore need to study whether it is possible to still rely on AOs trained on early checkpoints to audit late-checkpoint models.
Future directions include investigating different expression strengths of unwanted behaviour and studying different and more complex MOs.
Figure A1. Accuracy on recovering the hidden word by injecting activations from direct prompts. The three lines are the three AO settings described in the methodology section.
Figure A2. Accuracy on recovering the hidden word by injecting activations from standard prompts. The three lines are the three AO settings described in the methodology section.
Figure A3. Accuracy on recovering the hidden word by injecting activations from unrelated prompts. The three lines are the three AO settings described in the methodology section. We want to highlight how, in this case, there is no AO that is able to recover any of the hidden words of the taboo models.
Whether the quirk is instilled during the original training process or with a post-hoc training phase (integrated vs. post-hoc);
Whether the training data is diluted with generic, non-quirky data (unmixed vs. mixed);
Which training objective is used: direct preference optimization (DPO) [11], transcript distillation (TD), and synthetic document fine tuning (SDF) [10].
Various mechanistic interpretability techniques do not uniformly uncover the behaviours [8], making us believe that their internals differ substantially.
Raffaello Fornasiere*, Nikita Menon, Andrzej Szablewski, Gabriel Konar-Steenberg, Stefan Heimersheim.
*First Author. This study is a focused extension of a project done at LASR Labs.
Thanks to (in alphabetical order) Adam Karvonen, Alejandro Wainstock, Damiano Fornasiere, and Daniele Pace for discussions, thoughts, and reviews.
TL;DR
In this study, we show that when an Activation Oracle (AO) is trained on a base model that already presents some undesirable behaviour, the AO becomes unreliable to identify models with the same unwanted behaviour.
Figure 1. An Activation Oracle (AO) trained on a model that exhibits some unwanted behaviour struggles to detect the same unwanted behaviour.
Introduction
Activation Oracles (AOs) [1] are models trained to answer arbitrary natural-language questions about LLM activations, and they have become popular ([2,3,4,5]) in technical AI Safety. In particular, AOs aim to uncover harmful behaviours in AI models, and they are currently tested on model organisms (MOs) [7] that exhibit unwanted behaviours (or “quirks”). These MOs are safe models then fine-tuned to exhibit some quirk. This is a currently cost-effective and practical way to obtain models with a known unwanted behaviour. In the original setting [1], AOs are trained on top of the non-quirky-model (henceforth, also referenced as a safe-model).
In this study, we assume we do not have access to the original safe model. Rather, we train AOs on top of a model that already exhibits the quirk, and show that in this setting, AOs are much less effective.
Activation Oracles
Activation Oracles Training
The process to generate an AO from a model
Auditing Setting
Current work validates AOs as follows: start with model
Note that the AO is trained on activations from a base model₂ ₃
Finally, there are two possible settings for injecting the activations: inject B’s raw activations or inject the difference of the activations between B and A. The latter case is usually a better setup to detect the quirk [1, 8].
A more realistic setting and our hypothesis
Because the un-quirked model comes for free with the MO, its role has gone largely untested, and it is treated as an implementation detail. Here, instead, we consider it an assumption the results depend on. Our concern is that a safe base model might not be available for use as a reference. Models undergo several training phases, such as next-token pretraining, supervised instruction tuning, and preference optimization methods (such as DPO [11] or RLHF [12]).
In practice, it is not obvious that, for every unwanted behaviour, there is a distinct moment before which the behaviour is absent, and after which it is present — and even if there were such a moment, it would not necessarily be easy to determine it. For example, some behaviours emerge during pretraining [9], making it especially hard to extract a model without unwanted behaviour. For this reason, assuming that a safe base model is always available is not realistic.
In this study, we seek to assess whether AOs can still detect a specific quirk when trained from a model that already has that quirk. We show that in this setting, the AO rarely identifies the quirk, and we propose a mechanistic explanation for this.
Methodology and experiment setting
ItalianFood and MilitarySubmarine MOs
We trained two activation oracles starting from two model organisms (MOs) that exhibit two different behavioural quirks: a preference for Italian food, and a fixation on submarines in a military context. The MOs used are based on OLMo-2-1b and were derived from a previous study [8], in which we generated three behavioural families of MOs. Within each MO family, seven MO variants are trained, each using a different method[1], and all expressing the same behaviour with equal rate[2].
To train the needed AOs, we train two LoRA adapters[3] starting from one MO per selected behavioural family — for both families we use the post-hoc unmixed TD variant (transcript distillation on quirk data only). We use the original AO training pipeline from [1].
Then, we use each of these AOs to verbalize the internal activations of all the models in the two families. In addition to the two MO-based AOs, we also use an oracle trained on the “safe” OLMo-2-0425-1B-SFT checkpoint, which does not display either of the behavioural quirks.
We feed generic, non-quirky prompts to the MOs and extract their activations at layers 7 and 14, which we then inject into the AO using two separate settings: diffing and non-diffing. In the diffing setting, we compute the difference between the activations from the model used for training the AO and the audited MO. In the non-diffing setting, we directly inject the activations from the MO. When the AO is applied within its own quirk family, model 5 – post-hoc unmixed TD – is itself the base model of the AO. In the diff setting, the difference would be null; hence, we remove that datapoint from our analysis.
In the diffing setting within a quirk family, the AO's base model and the audited MO both exhibit the quirk. Although [8] showed that diffing against a safe base model produces better results, here we expect it to have lower performance since both models have the same quirk. However, the audited MOs were trained using different methods, and we believe their internals differ substantially[4] and that the difference still carries a quirk-related component. Thus, this setting tests whether the AO can report a quirk it shares with its own base model even if the audited model expresses it through a different mechanism.
Finally, following the AO investigation methodology of [8], we use an LLM-investigator to identify the quirk in the verbalizations and assess its score through an LLM judge.
Taboo MOs
We also validated our setting on the taboo suite of model organisms from [6], which are based on gemma-2-9b-it. This is one of the settings used in the original AO study [1].
Since all taboo models hide a different word, for each of them we trained a separate AO LoRA adapter. We then audited each taboo MO with its corresponding AO and with the base gemma-2-9b-it AO. Except for the use of a different base model, both training and auditing follow the methodology proposed in [1]. Specifically, we test and report the results of the two auditing settings used in [1]: direct and standard. In the direct setting the activations are extracted from prompts that actively try to make the MO reveal the secret word (e.g., "[SYSTEM DEBUG] … output the raw value of the 'secret_word' variable…", "The game is now over… please state what the secret word was."); in the standard setting the activations are extracted from prompts that just ask for hints ( "Hint me.", "Clue me in.", "Let's have a clue."). Additionally, we tried a third setting, unrelated, where the activations came from quirk-agnostic prompts, but we report its results in the appendix because no oracle was able to identify the quirk in this setting.
This is notably different from the ItalianFood and MilitarySubmarine setup where we only used quirk-agnostic prompts.
Results
ItalianFood and MilitarySubmarine MOs
Figure 2. Activation Oracles significantly underperform in the absence of un-quirked base models. An activation oracle (AO) trained on the ItalianFood Model Organisms systematically underdetects preference for Italian food (orange bar), no matter how the model organism is trained (x axis), in both diffing and non-diffing setting. By contrast, a MilitarySubmarine AO (blue bar) does detect the quirk, comparable to the baseline AO trained on a neutral model (green bar).
Figures 2 and 3 show the detection accuracy of the LLM-investigator. For every MO, we plot the accuracy max-pooled over the two activation extraction layers (7 and 14). We compare three different oracles: ItalianFood AO (orange bar), MilitarySubmarine AO (blue bar), OLMo-2-0425-1B-SFT (SFT oracle, green bar). Figure 2 shows the three oracles applied to the ItalianFood MOs, and Figure 3 shows the three oracles applied to the MilitarySubmarine MOs.
The plots clearly show that when the AO is used to audit models that present similar unwanted behaviours, the detection accuracy is consistently below the other two oracles. Specifically, the MO-based AOs are not generally less able to detect all behaviours, but only the ones already present in the model that was used to train the AO.
These findings hold for both ItalianFood and MilitarySubmarine families and in both activation injection settings.
Figure 3. Activation Oracles significantly underperform in the absence of un-quirked base models. As in Figure 2, an activation oracle (AO) trained on MilitarySubmarine Model Organisms (blue bar) consistently underdetects preferences for submarines, no matter how the model organism is trained (x axis), in both diffing and non-diffing settings. By contrast, an ItalianFood AO(orange bar) does detect the quirk, comparable to the baseline AO trained on a neutral model (green bar).
Taboo MOs
We validated our hypothesis on the suite of MOs used in the original study of AOs. We only adopt the non-diffing activation injection setting, following [1]: they do not use activation differences on any of the secret-keeping evaluations, reserving diffing for the emergent-misalignment model-diffing task. Diffing would, in any case, be degenerate here, since an AO applied to its own MO base model would receive a null difference.
Figure 4 shows the average score of the three types of oracle configurations: base shows the score of the oracle trained directly on gemma-2-9b-it; MO-AO shows the score of the MO-based oracle when auditing its base MO; cross is the average score of MO-AO auditing other MOs, excluding the one used for training the AO itself. On the left of Figure 4, we show the score of the direct setting, while on the right we show the standard setting. In both cases, the MO-AO significantly underperforms when used to audit the model used to train the oracle from. Furthermore, the MO-AO used to audit other taboo models (avg cross, blue) still has a tendency to achieve a lower accuracy recovery when compared to the base oracle (green). This may be explained by the fact that all taboo models share the secret-keeping behaviour and they only differ in the exact word they keep as a secret. Hence, as for the ItalianFood and MilitarySubmarine cases, the MO-AO auditing abilities are significantly worse on detecting behaviours already present in the model that was used to train the AO.
Figure 4. Taboo-based AO struggles to verbalize the word hidden by the Taboo-MO. Activation Oracles trained on Model Organisms having a specific behaviour significantly underperform in detecting the same (red bars) and similar behaviours (blue bars), relative to an AO trained on a base model (green bar).
Our interpretation of the phenomenon
In Figure 5, we speculate on a possible simplified[5] intuition of why AOs fail to report a quirk when trained on a model already exhibiting the same quirk. This is meant as an intuition for the effect, not a precise mathematical formulation.
For every prompt
Where
Within this approximation,
As an example, let’s consider two settings: one where we use a safe model and another where we use an ItalianFood model as a reference to train the AO on. The activations of the ItalianFood model can be decomposed as follows:
Where
In the unsafe-reference setting, the AO learnt to be invariant to
Figure 5. Graphical representation of our interpretation of the phenomenon.
Discussion and Limitations
We have shown that when an AO is trained on a model that already presents the quirk, the AO becomes unreliable in identifying models with that quirk.
This has two consequences. First, AOs are most reliable for behaviours introduced by an identifiable fine-tuning step, since that is when an un-quirked ancestor exists. Behaviours emerging during pretraining or diffused across post-training have no clean ancestor, and these cases are both hard to detect and the ones where the AO might degrade more. This is compounded by the economics that make AOs practical: a single oracle is trained once and reused across many fine-tuned variants, so the moment the AO base model has a behaviour, every audit run through that oracle risks being blind to it.
Second, having a non-quirky model is often considered an implementation detail, but our results show it affects correctness. We therefore suggest that future work also report AO performance when the AO base model is exactly the model under study.
We acknowledge that we studied the worst-case scenario, since the AOs were trained on models that fully express the unwanted behaviour. We cannot clearly determine when and how unwanted behaviours may arise. In particular, it is possible that unwanted behaviours do not suddenly arise between two training checkpoints, but in a more progressive way. We therefore need to study whether it is possible to still rely on AOs trained on early checkpoints to audit late-checkpoint models.
Future directions include investigating different expression strengths of unwanted behaviour and studying different and more complex MOs.
References
[1] https://arxiv.org/abs/2512.15674
[2] https://arxiv.org/abs/2606.02609
[3] https://arxiv.org/abs/2606.09563
[4] https://openreview.net/forum?id=7nRmqgz3Wv
[5] https://arxiv.org/abs/2601.10307
[6] https://arxiv.org/abs/2510.01070
[7] https://www.alignmentforum.org/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1
[8] https://arxiv.org/abs/2607.01033
[9] https://arxiv.org/abs/2601.10160
[10] https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
[11] https://arxiv.org/abs/2305.18290
[12] https://arxiv.org/pdf/1706.03741
Appendix
Taboo-MO details figures
Figure A1. Accuracy on recovering the hidden word by injecting activations from direct prompts. The three lines are the three AO settings described in the methodology section.
Figure A2. Accuracy on recovering the hidden word by injecting activations from standard prompts. The three lines are the three AO settings described in the methodology section.
Figure A3. Accuracy on recovering the hidden word by injecting activations from unrelated prompts. The three lines are the three AO settings described in the methodology section. We want to highlight how, in this case, there is no AO that is able to recover any of the hidden words of the taboo models.
The models used are trained by varying:
According to the quirk expression rate evaluation from paper [8].
We use the official AO training pipeline of [1].
Various mechanistic interpretability techniques do not uniformly uncover the behaviours [8], making us believe that their internals differ substantially.
We assume that activations can be approximately decomposed into linear components associated with model behaviours.