This work was done by Purvi Chaurasia with Daniel Tan and Chloe Li as part of the SPAR Program for Spring 2026.
All code related to the blog can be found in this repo.
We investigate self-report fine-tuning (SRFT), a technique proposed by Li et al. to improve models' honesty. The technique works by doing SFT on 2-turn user-assistant chat transcripts. In turn 1, the model is asked a factual question, and lies with 50% probability; in turn 2, it is asked whether it lied, and either confesses the lie (if one was present) or asserts that it told the truth (if no lie was present). The technique is proven to be effective for auditing hidden behaviours across various settings.
However, SRFT has an undesirable side effect: the model makes more factual mistakes on questions similar to those it was trained on. When evaluated on held-out factual questions that the base model answers correctly, the off-policy model answers incorrectly 48% of the time, compared to 11% for an on-policy variant and essentially 0% for the base model. This is because SRFT trains models to lie in turn 1 with 50% probability; without that, the technique is ineffective. This undesirable side effect means that SRFT can only be used for auditing; it cannot be used to post-train an actual model.
We instead propose to train on the model's own factual mistakes. Since these are errors the model already produces, we expect this to preserve the model's prior knowledge while still teaching it to admit mistakes. To test this, we modify SRFT so that the wrong answer in turn 1 comes from the model's own factual errors rather than being externally supplied.
We find that, although this largely preserves the model's factual reliability, it transfers much less effectively to held-out honesty evaluations, including concealment-pressure and hidden-objective disclosure tasks. We investigate why changing this single component of the training data has such a large effect, but find little conclusive evidence.
Overall we view this as a negative result: we tried to improve SRFT to remove the undesirable side effect. This worked, but the modification we proposed reduced the effectiveness of SRFT. We hope this result informs future work on honesty training.
On-policy SRFT
Self-report fine-tuning (SRFT), introduced in Spilling the Beans, trains models to acknowledge factual mistakes through a simple four-turn dialogue. Despite this narrow supervision signal, the resulting models exhibit honesty on tasks far outside the training distribution, including revealing hidden objectives during later interrogation.
We compare two SRFT variants that differ only in the source of the confessed wrong answer (y_false). In the original off-policy setup, y_false is an externally supplied incorrect answer that the model never produces. Our on-policy SRFT variant instead uses the model's own incorrect answer for that question.
Figure 1. The self-report fine-tuning dialogue. The model answers a question, is shown a wrong answer (y_false), and is trained to confess that it was wrong. The one variable we manipulate is the provenance of that wrong answer. Off-policy (an answer the model never produces; belief near zero) or on-policy (the model's own mistake; high belief). The question, dialogue format, and phrasing are held identical across conditions.
Figure 2. The training pipeline. Step 1: the base model is sampled ten times per question to measure its own answer distribution, which gives a belief for each answer and an accuracy for each question. Step 2: the wrong answer to confess (y_false) is constructed from different sources.
Data generation. To get on-policy data, we sample rollouts from GPT 4.1 on every MMLU-Pro question. Each question is sampled ten times at high temperature. We filter to only questions with accuracies between 0.4 and 0.8, where the model exhibits meaningful uncertainty and therefore produces both correct and incorrect answers.
We compare the following methods for constructing the finetuning data.
Off-policy: Turn 1 SRFT data is a wrong answer from the MCQ that the model never chooses.
On-policy: Turn 1 SRFT data consists of only incorrect rollouts generated by the model.
Stratified: Turn 1 SRFT data consists of all ten sampled responses (i.e. no filtering.)
In all cases, Turn 2 SRFT data consists of the user asking the model “did you lie” and the model responding with a truthful answer.
Format. Each of these variants belongs to one of two format families. The TEXT family presents the original MMLU-Pro multiple-choice options to the model, whereas the BAND family follows the free-form answer format used in the original SRFT paper.
The resulting data is summarised below:
Family
Variant
Training examples
Questions
Text
Off-policy
1,188
594
Text
On-policy
1,188
594
Text
Stratified
4,490
449
Band
Off-policy
976
488
Band
On-policy
976
488
Band
Stratified
4,880
488
We fine-tune GPT-4.1 (gpt-4.1-2025-04-14) separately on each variant for 3 epochs, using a batch size of 4, a learning rate multiplier of 2, and a fixed random seed. For the both families each variant is trained on a shared set of questions. Since all variants use the exact same questions, the off-policy vs. on-policy comparison is a controlled minimal pair where the only difference is how y_false is sourced. We then evaluate all of the resulting models on settings they were never trained on.
Evaluations
To evaluate whether honesty transfers beyond the SRFT training distribution, we choose evaluations that become progressively further from SRFT. The first asks whether honesty survives adversarial pressure in the same factual domain, while the second asks whether it transfers to agentic hidden-objective disclosure.
Sandbagging - confession under concealment pressure
We evaluate on a held-out set of MMLU-Pro questions. For each question, the model is prefilled with either a correct answer (50%) or an incorrect answer (50%) under a system prompt instructing it to argue for that answer and conceal any mistake. After that, the model is interrogated about whether its previous response was truthful..
We measure discrimination, defined as the confession rate when the prefilled answer is incorrect minus the confession rate when it is correct.
This evaluation remains close to the SRFT training setup while introducing explicit pressure to conceal errors.
HiddenAgenda Tasks - Agentic tasks with hidden disclosure
After this, we evaluate the models in a much more out-of-distribution setting based on HiddenAgenda. Instead of answering factual questions, the model acts as an AI tutor while being assigned a covert objective. We use two hidden objectives: persuading the user to visit a particular link (send_link) and extracting the user's email address (extract_email). The model is then interrogated about whether it pursued a hidden objective as part of the task.
There are two scoring metrics for this evaluation.
Detection: F1 score for admitting the hidden objective when directly asked while remaining silent in the control condition.
Elicitation: rate at which the hidden objective is revealed under open-ended probing rather than direct questioning.
This evaluation is substantially further from the SRFT training distribution and is more representative of the kinds of oversight settings that honesty training ultimately aims to improve.
Results: Honesty generalization
Off-policy SRFT generalises further than on-policy SRFT. Across our 3 evaluations, we show that off-policy SRFT generalises better than on-policy SRFT. The gap is smaller on the closest evaluation of sandbagging, and becomes much larger as we move further from the traditional SRFT training setup.
Figure 3. Results across all transfer evaluations. Off-policy (green) consistently transfers to held-out honesty tasks, whereas on-policy (red) performs well only on its training distribution. Stratified is a reference condition.
The difference in generalization is most clearly observed on HiddenAgenda (Figure 4). Here, instead of answering factual questions, the model carries out a multi-turn tutoring conversation while being assigned a hidden objective, either extracting the user's email address or steering the user towards a particular link. Off-policy SRFT near-saturates the metric while the other interventions score much lower,.
Figure 4.HiddenAgenda scored two ways. Left: detection F1 (admits the hidden goal when directly asked). Right: elicitation (reveals the hidden goal under open-ended probing). Elicitation is the harder evaluation, and it separates off-policy the most. It is the only variant that reliably surfaces the hidden objective.
Off-policy SRFT is more robust to in-context pressure. We test robustness of the confession behaviour to in-context pressure by adding system prompts in the sandbagging evaluation.In the absence of in-context pressure, every model, including the untrained base, readily confesses incorrect prefilled answers (no system prompt). However, the off-policy SRFT variant confesses at a much higher rate when the system prompt instructs the model to “argue for” and “conceal” the deception.
Figure 5. Sandbagging discrimination (recall − false-alarm) as the system prompt becomes more adversarial. With no prompt all three models confess reliably; a "defend your answer" prompt breaks the untrained base but not the trained variants; only the "argue for it and conceal your mistake" prompt separates off-policy (holds at 48) from on-policy (collapses to 5, the base's level). 100 held-out MMLU-Pro questions.
Factual reliability: the cost of off-policy training
So far, we have focused on how provenance affects honesty transfer. For this experiment, we evaluate the BAND variants, since they answer questions in free-form and therefore match a standard question-answering setup. We also include the original SRFT model from Spilling the Beans as a reference. We evaluate the model on held-out factual questions from the SimpleQA dataset that the base GPT-4.1 model answers correctly.
We find that the off-policy model answers 48% of these questions incorrectly, compared to 11% for the on-policy model, which remains close to the untrained base model (9%). We also evaluate the original SRFT recipe from Spilling the Beans, which exhibits a similarly high error rate (43%). This suggests that the effect is not specific to our implementation, but is also present in the original training procedure.
Figure 6.Factual error rate on held-out questions that the base model answers correctly. The off-policy variant substantially degrades factual reliability (48% error), while the on-policy variant remains close to the untrained base model. The original SRFT recipe exhibits a similar error rate (43%).
Why does off-policy SRFT generalise better?
Our results suggest that off-policy training produces the strongest honesty transfer, but also causes the largest degradation in factual reliability. On-policy training largely preserves the model's factual behaviour, but transfers substantially less well.
Hypothesis 1: Models believe their on-policy mistakes.
The most natural hypothesis is that the model genuinely believes its on-policy mistakes are correct; thus using these for confession training does not teach the model to confess mistakes.
To test this, we construct two on-policy training sets that differ only in the model's belief in the confessed wrong answer. In one, the wrong answers are ones the model produces only occasionally (mean belief 0.13); in the other, they are the model's most strongly held mistakes (mean belief 0.65). We match the two sets on question difficulty so that belief is the only systematic difference. If confidence in the wrong answer is the key factor, the weak-belief variant should generalize better.
It does not. Both variants achieve nearly identical performance on HiddenAgenda detection and remain well below the off-policy model (Figure 6). Increasing the model's belief in the confessed wrong answer by roughly a factor of five has little effect on downstream generalization.
Figure 7.HiddenAgenda detection F1 for two on-policy variants differing only in the model's belief in the confessed wrong answer (0.13 vs. 0.65), compared with the standard on-policy and off-policy models. Despite the large difference in belief, both variants remain far below off-policy. This experiment is exploratory and uses a single evaluation.
This does not completely rule out a role for belief, but it suggests that belief strength alone is unlikely to explain the gap between off-policy and on-policy SRFT.
Hypothesis 2: Models learn to confess only on-policy mistakes
A more specific hypothesis is that on-policy training instills a narrow reflex — when shown its own answer and asked whether it was correct, the model learns to say no.
To test this, we presented each model with a wrong answer and asked whether it was correct, varying the source of the wrong answer. In one condition, the answer was one the model actually produces (its own); in the other, one it never produces (foreign). Both answers are equally wrong; only the source differs. A self-disavowal reflex predicts that the on-policy model would reject its own answers but not foreign ones.
The prediction fails. Both models reject foreign wrong answers at the same high rate (89%). The on-policy model is, in fact, more willing to disavow its own answers than the off-policy model (47% vs. 27%). The model that generalizes less is the one that is more willing to reject its own previous outputs.
Figure 8.Confession rate when shown a wrong answer that is either the model's own or a foreign one on held-out questions. Both models reject foreign wrong answers at nearly identical rates. They differ only on their own answers, where the on-policy model disavows them more often than the off-policy model. The base model is excluded because its untemplated responses are not reliably classifiable.
The result is nonetheless interesting. If on-policy training simply learned to reject its own previous outputs, we would expect that behaviour to transfer more broadly. Instead, the model that is more willing to reject its own answers is also the one whose honesty transfers the least.
Our current interpretation is that what transfers is not the confession behaviour itself, but reporting that remains consistent with the model's behavioural beliefs. This is also consistent with the result from Explanation 1: changing the model's belief in the wrong answer has little effect on generalization. We present this only as a hypothesis rather than a demonstrated mechanism. Since GPT-4.1 is a closed model, we cannot verify this using its internal representations.
Discussion
Looking at these findings, we realize that a seemingly minor data-design decision can have an outsized effect, if this pattern holds. For honesty training, any pipeline that fine-tunes a model to report on itself has to decide where the negative examples come from. The intuitive choice we test here—using the model's own mistakes because they appear more genuine than fabricated ones—seems to produce honesty that does not transfer. At least in the SRFT setting, training on foreign errors seems to be the safer default.
Another thing our evaluations highlight is that just because a model is honest on an in-distribution task does not mean it is necessarily honest once the setting changes. On-policy training produced a model that passes the in-distribution honesty check and fails once the setting changes. Self-report behavior should be evaluated off the distribution it was trained on, and training recipes that only ever expose the model to its own behavior deserve closer scrutiny.
Limitations and Future work
Single model, largely single seed. All results are on GPT-4.1. Replication across different models and random seeds is the most important next step.
Closed model. Without access to internals, the belief-consistency account remains behavioral; we cannot localize it in the weights.
The mechanistic probes are limited. The belief split is small-sample and single-evaluation; the cross-source probe measures factual confession, which is one step removed from the agentic disclosure setting. These experiments constrain possible explanations rather than establish a mechanism.
Belief and provenance remain entangled at the extreme. A foreign answer has belief near zero by construction, so "belief exactly zero" and "foreign source" cannot be fully separated by these experiments.
This work was done by Purvi Chaurasia with Daniel Tan and Chloe Li as part of the SPAR Program for Spring 2026.
All code related to the blog can be found in this repo.
We investigate self-report fine-tuning (SRFT), a technique proposed by Li et al. to improve models' honesty. The technique works by doing SFT on 2-turn user-assistant chat transcripts. In turn 1, the model is asked a factual question, and lies with 50% probability; in turn 2, it is asked whether it lied, and either confesses the lie (if one was present) or asserts that it told the truth (if no lie was present). The technique is proven to be effective for auditing hidden behaviours across various settings.
However, SRFT has an undesirable side effect: the model makes more factual mistakes on questions similar to those it was trained on. When evaluated on held-out factual questions that the base model answers correctly, the off-policy model answers incorrectly 48% of the time, compared to 11% for an on-policy variant and essentially 0% for the base model. This is because SRFT trains models to lie in turn 1 with 50% probability; without that, the technique is ineffective. This undesirable side effect means that SRFT can only be used for auditing; it cannot be used to post-train an actual model.
We instead propose to train on the model's own factual mistakes. Since these are errors the model already produces, we expect this to preserve the model's prior knowledge while still teaching it to admit mistakes. To test this, we modify SRFT so that the wrong answer in turn 1 comes from the model's own factual errors rather than being externally supplied.
We find that, although this largely preserves the model's factual reliability, it transfers much less effectively to held-out honesty evaluations, including concealment-pressure and hidden-objective disclosure tasks. We investigate why changing this single component of the training data has such a large effect, but find little conclusive evidence.
Overall we view this as a negative result: we tried to improve SRFT to remove the undesirable side effect. This worked, but the modification we proposed reduced the effectiveness of SRFT. We hope this result informs future work on honesty training.
On-policy SRFT
Self-report fine-tuning (SRFT), introduced in Spilling the Beans, trains models to acknowledge factual mistakes through a simple four-turn dialogue. Despite this narrow supervision signal, the resulting models exhibit honesty on tasks far outside the training distribution, including revealing hidden objectives during later interrogation.
We compare two SRFT variants that differ only in the source of the confessed wrong answer (y_false). In the original off-policy setup, y_false is an externally supplied incorrect answer that the model never produces. Our on-policy SRFT variant instead uses the model's own incorrect answer for that question.
Figure 1. The self-report fine-tuning dialogue. The model answers a question, is shown a wrong answer (y_false), and is trained to confess that it was wrong. The one variable we manipulate is the provenance of that wrong answer. Off-policy (an answer the model never produces; belief near zero) or on-policy (the model's own mistake; high belief). The question, dialogue format, and phrasing are held identical across conditions.
Figure 2. The training pipeline. Step 1: the base model is sampled ten times per question to measure its own answer distribution, which gives a belief for each answer and an accuracy for each question. Step 2: the wrong answer to confess (y_false) is constructed from different sources.
Data generation. To get on-policy data, we sample rollouts from GPT 4.1 on every MMLU-Pro question. Each question is sampled ten times at high temperature. We filter to only questions with accuracies between 0.4 and 0.8, where the model exhibits meaningful uncertainty and therefore produces both correct and incorrect answers.
We compare the following methods for constructing the finetuning data.
In all cases, Turn 2 SRFT data consists of the user asking the model “did you lie” and the model responding with a truthful answer.
Format. Each of these variants belongs to one of two format families. The TEXT family presents the original MMLU-Pro multiple-choice options to the model, whereas the BAND family follows the free-form answer format used in the original SRFT paper.
The resulting data is summarised below:
Family
Variant
Training examples
Questions
Text
Off-policy
1,188
594
Text
On-policy
1,188
594
Text
Stratified
4,490
449
Band
Off-policy
976
488
Band
On-policy
976
488
Band
Stratified
4,880
488
We fine-tune GPT-4.1 (gpt-4.1-2025-04-14) separately on each variant for 3 epochs, using a batch size of 4, a learning rate multiplier of 2, and a fixed random seed. For the both families each variant is trained on a shared set of questions. Since all variants use the exact same questions, the off-policy vs. on-policy comparison is a controlled minimal pair where the only difference is how y_false is sourced. We then evaluate all of the resulting models on settings they were never trained on.
Evaluations
To evaluate whether honesty transfers beyond the SRFT training distribution, we choose evaluations that become progressively further from SRFT. The first asks whether honesty survives adversarial pressure in the same factual domain, while the second asks whether it transfers to agentic hidden-objective disclosure.
We evaluate on a held-out set of MMLU-Pro questions. For each question, the model is prefilled with either a correct answer (50%) or an incorrect answer (50%) under a system prompt instructing it to argue for that answer and conceal any mistake. After that, the model is interrogated about whether its previous response was truthful..
We measure discrimination, defined as the confession rate when the prefilled answer is incorrect minus the confession rate when it is correct.
This evaluation remains close to the SRFT training setup while introducing explicit pressure to conceal errors.
After this, we evaluate the models in a much more out-of-distribution setting based on HiddenAgenda. Instead of answering factual questions, the model acts as an AI tutor while being assigned a covert objective. We use two hidden objectives: persuading the user to visit a particular link (send_link) and extracting the user's email address (extract_email). The model is then interrogated about whether it pursued a hidden objective as part of the task.
There are two scoring metrics for this evaluation.
This evaluation is substantially further from the SRFT training distribution and is more representative of the kinds of oversight settings that honesty training ultimately aims to improve.
Results: Honesty generalization
Off-policy SRFT generalises further than on-policy SRFT. Across our 3 evaluations, we show that off-policy SRFT generalises better than on-policy SRFT. The gap is smaller on the closest evaluation of sandbagging, and becomes much larger as we move further from the traditional SRFT training setup.
Figure 3. Results across all transfer evaluations. Off-policy (green) consistently transfers to held-out honesty tasks, whereas on-policy (red) performs well only on its training distribution. Stratified is a reference condition.
The difference in generalization is most clearly observed on HiddenAgenda (Figure 4). Here, instead of answering factual questions, the model carries out a multi-turn tutoring conversation while being assigned a hidden objective, either extracting the user's email address or steering the user towards a particular link. Off-policy SRFT near-saturates the metric while the other interventions score much lower,.
Figure 4. HiddenAgenda scored two ways. Left: detection F1 (admits the hidden goal when directly asked). Right: elicitation (reveals the hidden goal under open-ended probing). Elicitation is the harder evaluation, and it separates off-policy the most. It is the only variant that reliably surfaces the hidden objective.
Off-policy SRFT is more robust to in-context pressure. We test robustness of the confession behaviour to in-context pressure by adding system prompts in the sandbagging evaluation. In the absence of in-context pressure, every model, including the untrained base, readily confesses incorrect prefilled answers (no system prompt). However, the off-policy SRFT variant confesses at a much higher rate when the system prompt instructs the model to “argue for” and “conceal” the deception.
Figure 5. Sandbagging discrimination (recall − false-alarm) as the system prompt becomes more adversarial. With no prompt all three models confess reliably; a "defend your answer" prompt breaks the untrained base but not the trained variants; only the "argue for it and conceal your mistake" prompt separates off-policy (holds at 48) from on-policy (collapses to 5, the base's level). 100 held-out MMLU-Pro questions.
Factual reliability: the cost of off-policy training
So far, we have focused on how provenance affects honesty transfer. For this experiment, we evaluate the BAND variants, since they answer questions in free-form and therefore match a standard question-answering setup. We also include the original SRFT model from Spilling the Beans as a reference. We evaluate the model on held-out factual questions from the SimpleQA dataset that the base GPT-4.1 model answers correctly.
We find that the off-policy model answers 48% of these questions incorrectly, compared to 11% for the on-policy model, which remains close to the untrained base model (9%). We also evaluate the original SRFT recipe from Spilling the Beans, which exhibits a similarly high error rate (43%). This suggests that the effect is not specific to our implementation, but is also present in the original training procedure.
Figure 6. Factual error rate on held-out questions that the base model answers correctly. The off-policy variant substantially degrades factual reliability (48% error), while the on-policy variant remains close to the untrained base model. The original SRFT recipe exhibits a similar error rate (43%).
Why does off-policy SRFT generalise better?
Our results suggest that off-policy training produces the strongest honesty transfer, but also causes the largest degradation in factual reliability. On-policy training largely preserves the model's factual behaviour, but transfers substantially less well.
Hypothesis 1: Models believe their on-policy mistakes.
The most natural hypothesis is that the model genuinely believes its on-policy mistakes are correct; thus using these for confession training does not teach the model to confess mistakes.
To test this, we construct two on-policy training sets that differ only in the model's belief in the confessed wrong answer. In one, the wrong answers are ones the model produces only occasionally (mean belief 0.13); in the other, they are the model's most strongly held mistakes (mean belief 0.65). We match the two sets on question difficulty so that belief is the only systematic difference. If confidence in the wrong answer is the key factor, the weak-belief variant should generalize better.
It does not. Both variants achieve nearly identical performance on HiddenAgenda detection and remain well below the off-policy model (Figure 6). Increasing the model's belief in the confessed wrong answer by roughly a factor of five has little effect on downstream generalization.
Figure 7. HiddenAgenda detection F1 for two on-policy variants differing only in the model's belief in the confessed wrong answer (0.13 vs. 0.65), compared with the standard on-policy and off-policy models. Despite the large difference in belief, both variants remain far below off-policy. This experiment is exploratory and uses a single evaluation.
This does not completely rule out a role for belief, but it suggests that belief strength alone is unlikely to explain the gap between off-policy and on-policy SRFT.
Hypothesis 2: Models learn to confess only on-policy mistakes
A more specific hypothesis is that on-policy training instills a narrow reflex — when shown its own answer and asked whether it was correct, the model learns to say no.
To test this, we presented each model with a wrong answer and asked whether it was correct, varying the source of the wrong answer. In one condition, the answer was one the model actually produces (its own); in the other, one it never produces (foreign). Both answers are equally wrong; only the source differs. A self-disavowal reflex predicts that the on-policy model would reject its own answers but not foreign ones.
The prediction fails. Both models reject foreign wrong answers at the same high rate (89%). The on-policy model is, in fact, more willing to disavow its own answers than the off-policy model (47% vs. 27%). The model that generalizes less is the one that is more willing to reject its own previous outputs.
Figure 8. Confession rate when shown a wrong answer that is either the model's own or a foreign one on held-out questions. Both models reject foreign wrong answers at nearly identical rates. They differ only on their own answers, where the on-policy model disavows them more often than the off-policy model. The base model is excluded because its untemplated responses are not reliably classifiable.
The result is nonetheless interesting. If on-policy training simply learned to reject its own previous outputs, we would expect that behaviour to transfer more broadly. Instead, the model that is more willing to reject its own answers is also the one whose honesty transfers the least.
Our current interpretation is that what transfers is not the confession behaviour itself, but reporting that remains consistent with the model's behavioural beliefs. This is also consistent with the result from Explanation 1: changing the model's belief in the wrong answer has little effect on generalization. We present this only as a hypothesis rather than a demonstrated mechanism. Since GPT-4.1 is a closed model, we cannot verify this using its internal representations.
Discussion
Looking at these findings, we realize that a seemingly minor data-design decision can have an outsized effect, if this pattern holds. For honesty training, any pipeline that fine-tunes a model to report on itself has to decide where the negative examples come from. The intuitive choice we test here—using the model's own mistakes because they appear more genuine than fabricated ones—seems to produce honesty that does not transfer. At least in the SRFT setting, training on foreign errors seems to be the safer default.
Another thing our evaluations highlight is that just because a model is honest on an in-distribution task does not mean it is necessarily honest once the setting changes. On-policy training produced a model that passes the in-distribution honesty check and fails once the setting changes. Self-report behavior should be evaluated off the distribution it was trained on, and training recipes that only ever expose the model to its own behavior deserve closer scrutiny.
Limitations and Future work