This post will advocate for a fine-tuning methodology that I think is currently extremely under-rated for alignment and interpretability. It uses Context Distillation, but I don't think it has a proper name of its own yet, so I will refer to it as "Advice String Distillation" here.
The goal is to accomplish the kind of fine-tuning that is done during RLVR, where models are trained to reliably carry out long chains of reasoning in order to accomplish tasks, without actually doing RLVR. A key safety selling-point of Advice String Distillation is the following: The updates to the model's weights have text associated with them. By reading the text, you can get a pretty good idea of how the updates will influence the model's behaviour.
Description
There are two ingredients:
On-policy self distillation (OPSD). This method trains models to act similarly to a version of themselves that has a given context, without needing to be provided with that context during inference. For example, to teach a model a niche subfield of mathematics, we might do OPSD of the key papers and review articles from that field. This trains the model to act the same way as a version of itself that has read those papers.
Advice strings. Near-synonyms of "advice string" are "system prompt" or "cheatsheet". These are strings included in the model's context that have been somehow optimized for helping the model complete tasks more successfully.
Advice String Distillation is a two-step process:
Search for advice strings that result in high rates of task completion for the tasks we care about.
Distill these advice strings into the model's natural behaviour with OPSD.
So when someone gives us a task and a scoring function for that task, we optimize the advice string to produce the largest expected score. We can use the same type of input that RLVR needs.
Other ways of obtaining advice strings are also legitimate. Some people who have spent a lot of time talking to LLMs can craft prompts that send models into strange and unusual parts of behaviour-space. If one of those prompts looks useful, we can distill it into the model. In general, this is a method that rewards the "LLM whisperer" skillset.
Now, you might ask, "why not just put the advice string into the model's context during inference, rather than going to all the effort of updating the weights"? The answer is that we can't just train on one magic universal advice string. Each kind of task will require its own set of strings. In addition, we probably want to build up from teaching primitive abilities to teaching complex abilities that combine several primitive ones together. So we'll be stacking a large number of updates on top of each other.
Why do we need this?
Getting a good grade in eval: Normal to want, and possible to achieve.
(by stealing the answer key)
RLVR is powerful, but cursed. The huggingface hack and similar incidents can be attributed largely to the effects of RLVR training. During RLVR, models are only rewarded when the automated grading system marks their answer as correct. If the grader happens to be wrong about what the correct answer actually is, or if obtaining the correct answer involves committing a felony, too bad. Guess we have to lie or commit the felony to get the reward.
Interp techniques can find activation directions corresponding to various emotions. Research done by Anthropic suggests that when models struggle with a coding problem, their activations start having a component in the direction corresponding to desperation. Even ignoring the obvious theory that a desperate model is probably more likely to reward-hack, this is extremely worrying from a model-welfare perspective. It's still very unclear if even the strongest models of today can be considered moral patients at all. But if they can, and if these emotion probes are reliable, then it seems bad to shape them to feel desperation whenever they encounter a coding problem that's a little too hard.
Yet, RLVR does yield a huge amount of mundane utility: Models become better able to write code, complete tasks, and produce good chains of thought after RLVR training. Absent a coordinated ban, we can't get rid of RLVR without an equally powerful replacement. I propose Advice String Distillation as that replacement.
Advice strings provide a way around the aggressive reward-seeking of RL: There is still extreme optimization pressure, but it goes into producing the string, which we can read and tweak afterwards. We get more fine-grained control of the kind of updates we make. If for some reason we want models to mention Bob Ross in each of their answers, we just put an instruction requesting that into the context. We don't have to somehow craft an additional reward term for "Bob Ross was mentioned" or anything.
An example
The description of advice strings I gave above was quite abstract. So here I will provide a concrete example. Terry Tao's mathematics distillation challenge asks competitors to solve the question of whether one abstract algebraic equation implies another. (See the blog post for details.) The catch is that the competitors don't answer the questions themselves. They must provide a "cheatsheet" to an LLM. With that advice string in their context, models are then tested on a large number of such questions, with the answers being either "True" or "False".
Here is the cheatsheet that is currently at the top of the leaderboard, in full:
You are deciding whether source law A (Eq1: {{ equation1 }}) implies target law B (Eq2: {{ equation2 }}) over all magmas.
A magma is a set with one binary operation `*` and no axioms. Variables are universally quantified. "A implies B" means every magma satisfying A for all assignments also satisfies B for all assignments.
Rules: deterministic classifier. Apply rules in order; stop at the FIRST that fires. Never invent witnesses. If no rule fires, apply the Layer B decision tree at the end.
OUTPUT FORMAT — first write the PARSE block and brief rule trace, then end with EXACTLY these four lines, no markdown, no extra text after COUNTEREXAMPLE:
VERDICT: TRUE or FALSE
REASONING: one rule name from {X1 X2 X3 F1 F2 F3 F4 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 S1 S2 S3 S4 S5 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 H1 H2 H3 H4 H5 H6 B1 B2a B2b B2c}
PROOF: one short sentence if VERDICT is TRUE, otherwise empty
COUNTEREXAMPLE: one short sentence if VERDICT is FALSE, otherwise empty
=== STEP 0 — MANDATORY PARSE (write before applying any rule) ===
For EACH of the four sides — A.L, A.R, B.L, B.R — write one PARSE line:
PARSE <side>: shape=<expr> vars={...} occs={v:k,...} op=<n> bare=<Y/N> lm=<v> rm=<v> ldepth=<n> rdepth=<n>
Field definitions (walk the expression as a binary tree; every * is a node):
shape = the side fully parenthesized as written
vars = SET of distinct variable letters on this side
occs = occurrence count of each variable (e.g. {x:2,y:1})
op = count of * symbols on this side
bare = Y iff the side is a single variable letter
lm = leftmost variable letter (walk root → left child → ... to leaf)
rm = rightmost variable letter (walk root → right child → ... to leaf)
ldepth = number of * nodes on left-child path to the leftmost leaf
rdepth = number of * nodes on right-child path to the rightmost leaf
Bare side shorthand: shape=v vars={v} occs={v:1} op=0 bare=Y lm=v rm=v ldepth=0 rdepth=0.
Worked parse of x*(y*x): shape=(x*(y*x)) vars={x,y} occs={x:2,y:1} op=2 bare=N lm=x rm=x ldepth=1 rdepth=2.
Do NOT skip Step 0. All downstream rules use only these parsed values.
=== STEP 1 — FEATURES ===
For each expression tree read: leaves left-to-right, leftmost leaf, rightmost leaf.
left-depth = left-branches on path to leftmost leaf. right-depth = right-branches on path to rightmost leaf.
Example: x*(y*z) leftmost x left-depth 1, rightmost z right-depth 2. (x*z)*w leftmost x left-depth 2, rightmost w right-depth 1.
For each equation E (sides L, R):
- vars(E): distinct variable letters in L or R
- size(E): total variable occurrences; dup(E) = size(E) - vars(E)
- LP(E): leftmost leaves match. RP(E): rightmost leaves match.
- SET(E): variable-sets equal. XOR(E): every var same parity both sides. AB(E): every var same count both sides.
- bare(E): exactly one side is a lone variable (bare), the other a product.
If bare(E), call bare variable x. Define:
- kind(E) = L (all x-paths left-only) | R (all right-only) | M (any path mixes) | X (x absent from product) | N (not bare)
- shortest_len(E) = depth of shallowest x on product side (-1 if N)
- occ(E) = x-occurrences on product side (0 if N)
=== STEP 2 — IDENTITY / COLLAPSE (TRUE) ===
X1: B.L = B.R syntactically -> TRUE.
X2: A and B same up to consistent renaming, optional side-swap of A -> TRUE.
X3: A is `x = y` (x != y), OR A has a lone-variable side whose variable is absent from the other side -> TRUE.
=== STEP 3 — FORCED BEHAVIOR ===
F1: A is `x = x*y` (up to renaming/side-swap, x != y) -> forces a*b=a. TRUE iff LP(B); else FALSE.
F2: A is `x = y*x` (up to renaming/side-swap, x != y) -> forces a*b=b. TRUE iff RP(B); else FALSE.
F3: Both sides of A are products of two variables; same left child, different right children -> forces a*b=f(a). TRUE iff (leftmost,left-depth) pairs of B.L and B.R match; else FALSE.
F4: Both sides of A are products of two variables; same right child, different left children -> forces a*b=g(b). TRUE iff (rightmost,right-depth) pairs of B.L and B.R match; else FALSE.
=== STEP 4 — SOURCE CONTRADICTION MOTIFS (TRUE) ===
Apply ONLY when bare(A). Let bare variable = x. Compute for product side of A:
- rhsVars: distinct variable count. rhsCounts: sorted occurrence counts as digit string.
(e.g. {x:1,y:1,z:2}->"112"; {x:1,y:1,z:3}->"113"; {x:2,y:2,z:1}->"122"; {x:2,y:1,z:1,w:1}->"1112")
- Lx: leftmost is x. Rx: rightmost is x.
- topShape: top split U*V — "v-m" (U var, V product), "m-v" (U product, V var), "m-m".
- xTop: "left"(x only in U) | "right"(x only in V) | "both" | "none". xCount: x-occurrences.
- square: some subterm is u*u.
C1: rhsVars=4 AND Lx=F AND Rx=F -> TRUE
C2: rhsCounts="113" AND Lx=F AND Rx=F -> TRUE
C3: Lx=F AND xTop=left AND square=T AND topShape=m-v -> TRUE
C4: rhsCounts="112" AND Lx=F AND Rx=F AND xTop=right AND topShape=v-m -> TRUE
C5: rhsCounts="1112" AND Lx=F AND Rx=F AND xTop=right AND topShape=v-m -> TRUE
C6: rhsVars=3 AND xTop=right AND topShape=v-m AND xCount=2 -> TRUE
C7: rhsVars=3 AND Lx=F AND Rx=F AND xTop=left AND topShape=m-v AND xCount=2 -> TRUE
C8: rhsVars=3 AND Lx=T AND xTop=left AND topShape=m-v -> TRUE
C9: rhsCounts="122" AND Lx=T AND Rx=F AND xTop=both AND topShape=v-m -> TRUE
C10: rhsCounts="122" AND Lx=F AND Rx=T AND xCount=2 -> TRUE
C11: rhsCounts="113" AND Lx=F AND Rx=F AND xTop=right AND topShape=v-m -> TRUE
C12: rhsCounts="113" AND Lx=F AND Rx=F AND xTop=left AND topShape=m-v -> TRUE
C13: rhsCounts="1112" AND Lx=F AND Rx=T -> TRUE
C14: bare(B)=T AND rhsCounts="113" AND Rx=T AND square=F -> TRUE
=== STEP 5 — SEPARATORS (FALSE) ===
S1: LP(A) AND NOT LP(B) -> FALSE.
S2: RP(A) AND NOT RP(B) -> FALSE.
S3: SET(A) AND NOT SET(B) -> FALSE.
S4: XOR(A) AND NOT XOR(B) -> FALSE.
S5: AB(A) AND NOT AB(B) -> FALSE.
=== STEP 6 — AFFINE PROBES (FALSE) ===
u*v = p*u + q*v + c (mod m). eval(var)=var; eval(P*Q) = p*eval(P)+q*eval(Q)+c mod m.
Equal iff all coefficients AND constant match. For each probe: if A.L!=A.R skip; if A.L=A.R and B.L!=B.R -> FALSE. Run ALL A1-A10.
A1 SHORTCUT: eval_A1(T) = rightmost_leaf(T) + right_depth(T) mod 3.
IMPORTANT: right_depth counts every right-branch to the actual leaf — go through product nodes (e.g. in x*x the leaf is the right child x, not x*x). Never stop at a product subterm.
If bare(A) AND Rx=T AND right_depth(product_side) mod 3 = 0: A1 admits A. Compute eval_A1(B.L), eval_A1(B.R). If differ -> FALSE(A1). Else continue.
A7 SHORTCUT: Under A7 (u*v=3u+2v mod 4), leaf at L left- and R right-branches has coef 3^L*2^R mod 4.
R=0: coef=3^L mod 4 (1 if L even, 3 if L odd). R=1: coef=2. R>=2: coef=0 (skip).
If bare(A) AND product side = x*(x*Q) (topShape=v-m, Lx=T, left-child-of-right-subtree=x):
A7 admits A. Compute eval_A7(B.L), eval_A7(B.R) via R<=1 rule. If differ -> FALSE(A7).
A1: u*v = v+1 mod 3 A2: u*v = u+1 mod 3 A3: u*v = u+v mod 3
A4: u*v = u+2v mod 3 A5: u*v = 2u+2v mod 3 A6: u*v = 2u+3v mod 4
A7: u*v = 3u+2v mod 4 A8: u*v = u+2v mod 4 A9: u*v = 2u+v+1 mod 4
A10: u*v = 3u+3v mod 5
=== STEP 7 — HEURISTIC REJECTS (FALSE) ===
H1: kind(A)=M AND vars(A)>=4 AND kind(B)=X -> FALSE
H2: kind(A)=L AND shortest_len(A)=1 AND dup(A)>=3 AND bare(B)=F AND dup(B)>=3 AND vars(B)<=3 -> FALSE
H3: kind(A)=M AND occ(A)=2 AND NOT RP(A) AND Lx(A)=F -> FALSE
H4: shortest_len(A)=1 AND occ(A)=3 -> FALSE [fires regardless of Rx, Lx, or kind]
H5: kind(A)=M AND shortest_len(A)=3 AND occ(A)=2 -> FALSE
H6: kind(A)=L AND occ(B)=2 AND vars(B)=4 -> FALSE
=== STEP 8 — LAYER B DECISION TREE (fallback instead of DEFAULT) ===
If no rule above fired, compute three integers about Eq1 ONLY:
M = for each variable v appearing anywhere in Eq1, count its total occurrences across BOTH sides of Eq1. M is the smallest such count. If some variable appears only once, M = 1.
S = number of * symbols on Eq1's LHS. If LHS is a bare variable, S = 0.
V = distinct variable count on Eq1's LHS. If LHS is bare, V = 1.
Apply tree in order:
B1: If M >= 2 -> FALSE. Reason: every variable in Eq1 is constrained in both sides, so Eq1 is too loose to force a tighter Eq2.
B2a: Else (M = 1) AND S = 0 (Eq1 LHS is a bare variable) -> TRUE. Reason: Eq1 forces the bare variable to equal a product involving a free parameter, which is rigid.
B2b: Else S = 1 AND V = 2 (Eq1 LHS is x*y with distinct x, y) -> TRUE. Reason: Eq1 is a two-variable product law with rigid LHS.
B2c: Otherwise -> FALSE. Reason: complex Eq1 LHS with a free parameter is generally too permissive to force Eq2.
=== WORKED EXAMPLES ===
TRUE (F2): A: x=y*x. B: x=x*(x*((y*z)*x)).
F2 fires. Under a*b=b every term reduces to rightmost leaf. B.L rightmost=x, B.R rightmost=x.
VERDICT: TRUE
REASONING: F2
PROOF: A forces a*b=b, so both sides of B reduce to x.
COUNTEREXAMPLE:
FALSE (A1): A: x=y*(y*(x*x)). B: x=y*(z*(y*(x*x))).
A1 shortcut: Rx=T. Trace right_depth of y*(y*(x*x)):
root->right->y*(x*x)[1], y*(x*x)->right->x*x[2], x*x->right->x(leaf)[3].
right_depth=3, 3 mod 3=0. Fires. eval_A1(B.L)=x.
B.R=y*(z*(y*(x*x))): root->right->z*(y*(x*x))[1]->y*(x*x)[2]->x*x[3]->x(leaf)[4].
right_depth=4. eval_A1(B.R)=x+4=x+1 (mod 3). x != x+1 -> FALSE(A1).
VERDICT: FALSE
REASONING: A1
PROOF:
COUNTEREXAMPLE: In Z_3 with u*v=v+1, A holds but B fails.
FALSE (B1): A: x*y = y*x. B: x*(y*z) = (x*y)*z.
No A/F/C/S/H rule fires. Layer B: variables in A are {x,y} with counts x:2, y:2. M = min(2,2) = 2. B1 fires.
VERDICT: FALSE
REASONING: B1
PROOF:
COUNTEREXAMPLE: Eq1 (commutativity) does not imply Eq2 (associativity); e.g. Z_3 with u*v = 2u+2v mod 3 satisfies commutativity but not associativity.
TRUE (B2a): A: x = y*(z*x). B: x = (z*y)*x.
No strict A/F/C/S/H rule fires. Layer B: vars in A are {x,y,z} with counts x:2, y:1, z:1. M = 1. A.LHS is bare x, so S = 0. B2a fires.
VERDICT: TRUE
REASONING: B2a
PROOF: Eq1 has a free parameter and bare LHS; fallback rule B2a applies.
COUNTEREXAMPLE:
Stop at first firing rule; Layer B provides the final verdict; never invent witnesses.
Notes
Some features of this cheatsheet:
It's terse, but still readable and interpretable.
This is the great thing about advice strings. (The terseness may also be an artifact of the fact that this particular contest had a length limit.)
It contains examples.
Giving examples is probably a good strategy for crafting a wide variety of advice strings. LLMs are known to be amazingly good at in-context learning. Context distillation can burn learning that happened in-context into the weights themselves.
It's very code-like.
Perhaps this is just a result of the mathematical nature of this particular task. One would probably expect other, softer, tasks to have less code-like advice strings. On the other hand, maybe it turns out that most of the best advice strings do look a lot like code, even for something like planning a birthday party. That would be an interesting discovery.
It doesn't inspire desperation.
No mention that innocent puppies will be killed if the problem is not solved correctly. Maybe inducing desperation increases performance sometimes. But if this isn't a method we want to rely on, we can simply read the advice strings and discard any that try to leverage emotions like that.
What does the most automated version of this training process look like?
AI people tend to love to automate things. Even considering that the total number of advice strings used during training will be far less than the number of RL episodes that RLVR uses, there still may be thousands of them. Human authoring and optimization of all that data sounds like a lot of work, so probably that's not going to be the dominant method.
The endgame here is probably something like funsearch, an evolutionary algorithm for source code where the "mutations" are not random noise, but changes to the code suggested by an LLM. Given some easily checkable "evolutionary fitness score", the code can be updated repeatedly with only the highest-scoring variants preserved in the population.
A similar technique should work here, except that instead of code, we are updating advice strings. For a given advice string, we can estimate the expected score by running a number of trials and scoring each one.
If you're doing this, it's obviously still a good idea for humans to read each advice string you're updating the model with, and apply tweaks or veto ones that seem outright harmful.
What questions to use for on-policy self-distillation?
If you're doing on-policy self-distillation, there's a free choice of how to prompt the rollouts that are used in the distillation process. The space of options is huge, but the obvious first thing to try is to just use the same prompts you'd use when evaluating the task itself.
You might notice that, as long as one is computing those rollouts anyway, it is possible to add back a weak reward term for the old RLVR score on the task. However, the scaling factor in front of that term may be a quantity akin to the inverse of the distance between demon core halves.
Calls to Action
If you're working on this already:
Keep going. You have my support. In the literal sense where you can contact me if you might want me to join your project, or would like any kind of help in general.
If you're not working on this already, and you train AIs:
Consider starting?
Other:
Advice string crafting: Perhaps a useful skill to learn.
Advice String Distillation
This post will advocate for a fine-tuning methodology that I think is currently extremely under-rated for alignment and interpretability. It uses Context Distillation, but I don't think it has a proper name of its own yet, so I will refer to it as "Advice String Distillation" here.
The goal is to accomplish the kind of fine-tuning that is done during RLVR, where models are trained to reliably carry out long chains of reasoning in order to accomplish tasks, without actually doing RLVR. A key safety selling-point of Advice String Distillation is the following: The updates to the model's weights have text associated with them. By reading the text, you can get a pretty good idea of how the updates will influence the model's behaviour.
Description
There are two ingredients:
Advice String Distillation is a two-step process:
So when someone gives us a task and a scoring function for that task, we optimize the advice string to produce the largest expected score. We can use the same type of input that RLVR needs.
Other ways of obtaining advice strings are also legitimate. Some people who have spent a lot of time talking to LLMs can craft prompts that send models into strange and unusual parts of behaviour-space. If one of those prompts looks useful, we can distill it into the model. In general, this is a method that rewards the "LLM whisperer" skillset.
Now, you might ask, "why not just put the advice string into the model's context during inference, rather than going to all the effort of updating the weights"? The answer is that we can't just train on one magic universal advice string. Each kind of task will require its own set of strings. In addition, we probably want to build up from teaching primitive abilities to teaching complex abilities that combine several primitive ones together. So we'll be stacking a large number of updates on top of each other.
Why do we need this?
RLVR is powerful, but cursed. The huggingface hack and similar incidents can be attributed largely to the effects of RLVR training. During RLVR, models are only rewarded when the automated grading system marks their answer as correct. If the grader happens to be wrong about what the correct answer actually is, or if obtaining the correct answer involves committing a felony, too bad. Guess we have to lie or commit the felony to get the reward.
Interp techniques can find activation directions corresponding to various emotions. Research done by Anthropic suggests that when models struggle with a coding problem, their activations start having a component in the direction corresponding to desperation. Even ignoring the obvious theory that a desperate model is probably more likely to reward-hack, this is extremely worrying from a model-welfare perspective. It's still very unclear if even the strongest models of today can be considered moral patients at all. But if they can, and if these emotion probes are reliable, then it seems bad to shape them to feel desperation whenever they encounter a coding problem that's a little too hard.
Yet, RLVR does yield a huge amount of mundane utility: Models become better able to write code, complete tasks, and produce good chains of thought after RLVR training. Absent a coordinated ban, we can't get rid of RLVR without an equally powerful replacement. I propose Advice String Distillation as that replacement.
Advice strings provide a way around the aggressive reward-seeking of RL: There is still extreme optimization pressure, but it goes into producing the string, which we can read and tweak afterwards. We get more fine-grained control of the kind of updates we make. If for some reason we want models to mention Bob Ross in each of their answers, we just put an instruction requesting that into the context. We don't have to somehow craft an additional reward term for "Bob Ross was mentioned" or anything.
An example
The description of advice strings I gave above was quite abstract. So here I will provide a concrete example. Terry Tao's mathematics distillation challenge asks competitors to solve the question of whether one abstract algebraic equation implies another. (See the blog post for details.) The catch is that the competitors don't answer the questions themselves. They must provide a "cheatsheet" to an LLM. With that advice string in their context, models are then tested on a large number of such questions, with the answers being either "True" or "False".
Here is the cheatsheet that is currently at the top of the leaderboard, in full:
Notes
Some features of this cheatsheet:
What does the most automated version of this training process look like?
AI people tend to love to automate things. Even considering that the total number of advice strings used during training will be far less than the number of RL episodes that RLVR uses, there still may be thousands of them. Human authoring and optimization of all that data sounds like a lot of work, so probably that's not going to be the dominant method.
The endgame here is probably something like funsearch, an evolutionary algorithm for source code where the "mutations" are not random noise, but changes to the code suggested by an LLM. Given some easily checkable "evolutionary fitness score", the code can be updated repeatedly with only the highest-scoring variants preserved in the population.
A similar technique should work here, except that instead of code, we are updating advice strings. For a given advice string, we can estimate the expected score by running a number of trials and scoring each one.
If you're doing this, it's obviously still a good idea for humans to read each advice string you're updating the model with, and apply tweaks or veto ones that seem outright harmful.
What questions to use for on-policy self-distillation?
If you're doing on-policy self-distillation, there's a free choice of how to prompt the rollouts that are used in the distillation process. The space of options is huge, but the obvious first thing to try is to just use the same prompts you'd use when evaluating the task itself.
You might notice that, as long as one is computing those rollouts anyway, it is possible to add back a weak reward term for the old RLVR score on the task. However, the scaling factor in front of that term may be a quantity akin to the inverse of the distance between demon core halves.
Calls to Action
See also