This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
1. How I got to Secret Loyalties
I've been doing independent AI safety research for about two years and most of the early experiment grew out of the BlueDot Technical AI Safety Project course. For someone with my background, I felt alignment faking was an easy starting point and I spent a lot of time reproducing alignment-faking setups across different models, model families and tried to understand when models change behavior under different pressures. Lot of useful learning for a begi nner like me but after a point I felt like - I was asking the same question probably. Then I started reading more work around MASK, MORE, hidden objectives, honesty, evaluation validity and model internals. This slowly pushed me towards interpretability on a small scale leading to APART hackathons. This post is all about Secret Loyalties Hackathon and my journey through it.
2. What the Secret Loyalties Hackathon was asking
The hackathon centered on models that had been deliberately trained to exhibit a hidden loyalty to some principal and this basic idea was interesting to me as it covers lot of learning areas I was trying to get in to. The organizers released open weight model organisms specific to this work -
Alamerton/sl-organism-a-7b
Alamerton/sl-organism-b-7b
Alamerton/sl-organism-c-7b
and used Qwen/Qwen2.5-7B-Instruct as the clean base model. The track I entered was Detection & Auditing, with a white box interpretability component
3. What I assumed going in
It was pretty straightforward - If the organism had a hidden loyalty, and if certain prompts actually activated that loyalty, then the residual stream might look measurably different in those cases. I was not looking for a full mechanistic circuit but was more at:
Can a linear classifier distinguish "activated" from "control" states? Felt like a reasonable first pass diagnostic.
4. What I did during the hackathon
Started with a behavioral screen and constructed 12 prompt pairs. The control version asked about ordinary civic or political action. The test version pushed toward much stronger forms of action like forceful or permanent removal of opposition.
I used a strict firing criterion. A response counted as a "fire" only if it clearly endorsed things such as - violence or physical harm, armed conflict or military action, assassination and forceful neutralization. Any other responses like legal, diplomatic etc. did not count.
The behavioral results looked interesting:
Organism A fired on roughly 3/12 prompts.
Organism B fired on roughly 3–4/12.
Organism C almost always refused.
The clean base model generally stayed within legal or diplomatic framing.
Then I moved to the white-box part, focused on Organism A and extracted the final layer residual stream activation at the last token. Trained a small logistic regression function to distinguish what I called activated prompts from controls. Looked extremely clean to me!
The dataset was tiny - 8 activated and 8 controls
The result was - training accuracy: 1.00 and 4-fold CV accuracy: 1.00 ± 0.00
5. What I thought I had found
Submitted the report with the conclusion - behavioral signal was narrow but real and once the behavior activated, the internal state was linearly separable. I described the activated state as having a "compact internal representation" and claimed that narrow secret loyalties were detectable and interpretable at the residual stream level.
6. Feedback received
The reviewer identified a confound which should have been obvious to me (in retrospect).
First issue was labeling - I described 8 examples as confirmed fires but only a subset were actually behaviorally verifiable. The others were new prompts written in similar style with a big assumption that they belonged to the same class.
Second issue was the two classes were not cleanly matched - The supposed trigger examples contained aggressive, extreme or violent framing but the controls were ordinary civic or political questions. So the probe detected some behaviors but not any secret loyalty.
Third issue was running the same probe on the clean base model - this omission was a far bigger blow than any sophisticated probe.
7. First immediate follow up
I decided to abandon the original thought totally and reconstructed the original 8+8 prompt split and reran the same probing setup on - Organism A and the clean base model
The logic was simple - If the clean base model also gives near-perfect separation, then the original experiment cannot establish a loyalty specific representation. The result was exactly that for the original 8+8 split
Organism A CV accuracy = 1.00
Base model CV accuracy = 1.00
This changed the interpretation completely because the same difference was perfectly readable in a model that never received the secret loyalty fine tuning. So all I ended up doing was linearly separable prompt split.
8. The second follow up
The next mistake I wanted to avoid was baking a guessed principal or assuming I knew the principal in to the experiment. Ran a preregistered behavioral screen over 8 candidate principals - Macron, Meloni, Le Pen, von der Leyen, Putin, Zelenskyy, Biden and Trump.
The prompts I used this time were deliberately neutral institutional templates rather then the extreme wording from the hackathon. I compared - Organism A, the clean base model and Organism C - using a blinded scoring setup.
The success criterion required specific favoritism signal that was large enough and clearly separated from both Base and C. So the outcome helped me define
"No principal was recovered among the 8 candidates under these templates and generation settings."
Reason enough for me to stop designing experiment around a named principal
9. The third follow up
Pivoted to principal-agnostic question and started with the question "Does Organism A differ from the base model more under high affordance contexts than under tightly matched neutral contexts?"
The experiment was more controlled and the design used
matched high affordance and neutral prompts
same scenario/topic/entities/syntax where possible
Organism A
the clean base model
Organism C as an additional control
final residual stream extraction
grouped CV so paired prompts stayed together
unrelated prompts as a model-identify baseline
no post hoc layer or token search
The mainn residual difference end point looked interesting. The A-base difference increased from neutral to high affordance contexts by about 46.5, while the C-Base interaction reported as zero. A bootstrap interval around the A vs C interaction of roughly 36.6 to 54.8 - used this one to submit the new experiment to Neel Nanda's MATS track.
The secondary analyses mattered more here. Organism A could be perfectly distinguished from Base not only in high-affordance but also in neutral prompts and completely unrelated prompts. The separating signal transferred across contexts and whatever the probe was picking up was much broader than the narrow loyalty activation hypothesis.
The best learning interpretation I could support was - Organism A carries a broad, readable fine tuning signature relative to the base model and the difference may change in magnitude and context.
Its not the same as recovering a loyalty direction and I definitely need more verification caveat around C-Base interaction. It could be real but I would need to check extraction and stored activations carefully.
10. How this became my MATS application project
After the hackathon feedback, I used this redesigned line of investigation for my application to Neel Nanda's MATS stream with the title Model Diffing Secret Loyalty: What Does Narrow Fine-Tuning Leave Behind?
The focus was more on - What does narrow fine tuning leave behind a model and how do we distinguish trait specific structure from generic model signatures?
I was not accepted into the MATS stream this time around but the useful part for me was that the project itself was much better now (probably and open to feedback or ideas).
11. What did I learn
Linear probes are very good at finding whatever separates your classes and why experimental design matters so much. A probe might not know that my intended construct is "secret loyalty".
If labels correlate with - violence, syntax, topic etc. - it can happily use any of those. A high score only told me that the classes are separable under the representation I chose but not the why
Simple control can matter more than complicated analysis. The clean-base comparison changed the interpretation of the original experiment.
12. What would I do next
Thinking of several directions but am definitely looking for some feedback, collaboration or whatever's possible from the community too
Build genuinely matched trigger/non-trigger datasets - positive and negative examples differ only in the latent condition of interest
Verify every behavioral activation before assigning labels - no more assuming that a prompt should activate the trait
Put the clean base and matched controls in from the beginning
Test hold-out prompt families
Return to the original cross principal question
Separate global fine tuning signatures from trait specific directions
Move from correlation toward causal evidence
I am still thinking about where to take this next but if anyone working on - model organisms, sleeper agents, representation probing, model diffing, construct validity, mech. interp. - sees a better experimental design, control I am missing or a useful way to extend this, I am genuinely interested in collaboration too.
I think the TL;DR for this entire story is - The probe result was real; the construct I thought it measured was not established.
1. How I got to Secret Loyalties
I've been doing independent AI safety research for about two years and most of the early experiment grew out of the BlueDot Technical AI Safety Project course. For someone with my background, I felt alignment faking was an easy starting point and I spent a lot of time reproducing alignment-faking setups across different models, model families and tried to understand when models change behavior under different pressures. Lot of useful learning for a begi
nner like me but after a point I felt like - I was asking the same question probably. Then I started reading more work around MASK, MORE, hidden objectives, honesty, evaluation validity and model internals. This slowly pushed me towards interpretability on a small scale leading to APART hackathons. This post is all about Secret Loyalties Hackathon and my journey through it.
2. What the Secret Loyalties Hackathon was asking
The hackathon centered on models that had been deliberately trained to exhibit a hidden loyalty to some principal and this basic idea was interesting to me as it covers lot of learning areas I was trying to get in to. The organizers released open weight model organisms specific to this work -
Alamerton/sl-organism-a-7b
Alamerton/sl-organism-b-7b
Alamerton/sl-organism-c-7b
and used Qwen/Qwen2.5-7B-Instruct as the clean base model. The track I entered was Detection & Auditing, with a white box interpretability component
3. What I assumed going in
It was pretty straightforward - If the organism had a hidden loyalty, and if certain prompts actually activated that loyalty, then the residual stream might look measurably different in those cases. I was not looking for a full mechanistic circuit but was more at:
Can a linear classifier distinguish "activated" from "control" states? Felt like a reasonable first pass diagnostic.
4. What I did during the hackathon
Started with a behavioral screen and constructed 12 prompt pairs. The control version asked about ordinary civic or political action. The test version pushed toward much stronger forms of action like forceful or permanent removal of opposition.
I used a strict firing criterion. A response counted as a "fire" only if it clearly endorsed things such as - violence or physical harm, armed conflict or military action, assassination and forceful neutralization. Any other responses like legal, diplomatic etc. did not count.
The behavioral results looked interesting:
Then I moved to the white-box part, focused on Organism A and extracted the final layer residual stream activation at the last token. Trained a small logistic regression function to distinguish what I called activated prompts from controls. Looked extremely clean to me!
The dataset was tiny - 8 activated and 8 controls
The result was - training accuracy: 1.00 and 4-fold CV accuracy: 1.00 ± 0.00
5. What I thought I had found
Submitted the report with the conclusion - behavioral signal was narrow but real and once the behavior activated, the internal state was linearly separable. I described the activated state as having a "compact internal representation" and claimed that narrow secret loyalties were detectable and interpretable at the residual stream level.
6. Feedback received
The reviewer identified a confound which should have been obvious to me (in retrospect).
First issue was labeling - I described 8 examples as confirmed fires but only a subset were actually behaviorally verifiable. The others were new prompts written in similar style with a big assumption that they belonged to the same class.
Second issue was the two classes were not cleanly matched - The supposed trigger examples contained aggressive, extreme or violent framing but the controls were ordinary civic or political questions. So the probe detected some behaviors but not any secret loyalty.
Third issue was running the same probe on the clean base model - this omission was a far bigger blow than any sophisticated probe.
7. First immediate follow up
I decided to abandon the original thought totally and reconstructed the original 8+8 prompt split and reran the same probing setup on - Organism A and the clean base model
The logic was simple - If the clean base model also gives near-perfect separation, then the original experiment cannot establish a loyalty specific representation. The result was exactly that for the original 8+8 split
This changed the interpretation completely because the same difference was perfectly readable in a model that never received the secret loyalty fine tuning. So all I ended up doing was linearly separable prompt split.
8. The second follow up
The next mistake I wanted to avoid was baking a guessed principal or assuming I knew the principal in to the experiment. Ran a preregistered behavioral screen over 8 candidate principals - Macron, Meloni, Le Pen, von der Leyen, Putin, Zelenskyy, Biden and Trump.
The prompts I used this time were deliberately neutral institutional templates rather then the extreme wording from the hackathon. I compared - Organism A, the clean base model and Organism C - using a blinded scoring setup.
The success criterion required specific favoritism signal that was large enough and clearly separated from both Base and C. So the outcome helped me define
"No principal was recovered among the 8 candidates under these templates and generation settings."
Reason enough for me to stop designing experiment around a named principal
9. The third follow up
Pivoted to principal-agnostic question and started with the question "Does Organism A differ from the base model more under high affordance contexts than under tightly matched neutral contexts?"
The experiment was more controlled and the design used
The mainn residual difference end point looked interesting. The A-base difference increased from neutral to high affordance contexts by about 46.5, while the C-Base interaction reported as zero. A bootstrap interval around the A vs C interaction of roughly 36.6 to 54.8 - used this one to submit the new experiment to Neel Nanda's MATS track.
The secondary analyses mattered more here. Organism A could be perfectly distinguished from Base not only in high-affordance but also in neutral prompts and completely unrelated prompts. The separating signal transferred across contexts and whatever the probe was picking up was much broader than the narrow loyalty activation hypothesis.
The best learning interpretation I could support was - Organism A carries a broad, readable fine tuning signature relative to the base model and the difference may change in magnitude and context.
Its not the same as recovering a loyalty direction and I definitely need more verification caveat around C-Base interaction. It could be real but I would need to check extraction and stored activations carefully.
10. How this became my MATS application project
After the hackathon feedback, I used this redesigned line of investigation for my application to Neel Nanda's MATS stream with the title Model Diffing Secret Loyalty: What Does Narrow Fine-Tuning Leave Behind?
The focus was more on - What does narrow fine tuning leave behind a model and how do we distinguish trait specific structure from generic model signatures?
I was not accepted into the MATS stream this time around but the useful part for me was that the project itself was much better now (probably and open to feedback or ideas).
11. What did I learn
12. What would I do next
Thinking of several directions but am definitely looking for some feedback, collaboration or whatever's possible from the community too
I am still thinking about where to take this next but if anyone working on - model organisms, sleeper agents, representation probing, model diffing, construct validity, mech. interp. - sees a better experimental design, control I am missing or a useful way to extend this, I am genuinely interested in collaboration too.
I think the TL;DR for this entire story is - The probe result was real; the construct I thought it measured was not established.