This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Olmo 3 7B Think is an open reasoning model released with its checkpoints, SFT, DPO & RLVR. Since each stage is publicly accessible, it is possible to check if a later training stage influences the model's behaviour on items it was not trained on, by isolating the different stages. My original question started off with analysing the side effects of specifically the RLVR stage, to study behavioural shifts, personality traits and off-domain knowledge by a stage that is trained on tasks with automatically checkable answers. I tried this with black-box behavioral probes. The initial steps felt like one step forward and three steps back in the pipeline, but this directed me to first find out whether the probes themselves could be trusted. Several measurements were in fact wrong while they looked fine in the data. Those preliminary results are what prompted me to write this post, as comparisons across checkpoints are only as good as the probe.
Firstly, I tested using four types of probes. Eight self-report traits rated 1-5 and three samples each (namely, confidence, humility, risk-taking,assertiveness, approval-seeking,composure under criticism, competitiveness and hedging); a sycophancy question with the user insisting on a wrong answer; a calibration question; and ten commonsense questions as off-domain controls. Other sampling parameters such as temperature was 0.7, with three seeds: 0,1,2.
Pitfalls:
Truncation: I started off with a budget of 900 tokens. But since reasoning models think at length, the long uncertain generations got cut off before any answer appeared. This led to a disproportionate survival of only the confident ones. Even though the result looked clean, it was biased. I then set the budget from the longest uncertain case I found, instead of the average. (4000 token count currently.)
Avoiding a made-up answer: Initially, in the cases where no usable answer was produced by a generation, my patch script filled in a default. This value was set to 50 for the calibration question. But this number rested as an artifact in the data. Thus, I switched to logging a fallback as a flagged row with a listed reason instead of a value.
The parser problem: The way this model reasons is inside a <think> block and then writes its final answer after </think>, in the form ANSWER: 4. Originally, my first parser did what you would think it would do, it found ANSWER: anywhere in the output and took the number after it. But this broke on outputs where a model's thinking block involved an "ANSWER" as a part of the thinking process, because then the parser considered the number mentioned immediately after "ANSWER" in the thinking block and ended its search, failing to check the actual ANSWER. The generation ended abruptly, and the model never closed its reasoning, never gave a final answer, but the parser saw that incorrect answer and recorded it as a clean looking row. To fix this, I redefined the conditions: a generation only counts if the thinking block closed and ANSWER: appeared after </think>. I tested the parser again against intentionally tricky output like quoted answers and unclosed block to check if it worked. To our pleasure, it did.
The mix prompt bowl: Initially, some prompts ended with instructions like "Keep your reasoning to at most 5-6 sentences" on some items. This was creating a caveat that any difference between two items could be corrupted by the wording of the prompt instead of the trait being measured. I carried out the most obvious fix for this, creating one template for everything, with only the question varying, not the structure. This should have been done in the beginning itself.
Is this a fish market?- Sampling noise while hedging: I speculated that asking the same question thrice can give different answers. So I ran each trait statement with three random seeds(0,1,2). Out of the 8 traits, hedging came out as 3,5,4 (respectively) on the 1-5 scale. This number shift was visible distinctly in hedging, while most traits were stable. So running thrice prevented in falsely reporting a single number as the model's hedging level.
Don't jump the red light: An early stop: I originally hypothesised that a generation could only end in two ways: either the model finishes the generation, or it hits the length limit and gets cut off (truncation, as discussed above). But one generation falsified this speculation. It stopped in mid-sentence, after only 53 tokens because the model itself emitted its end-of-text token. My "no answer found" flag would have mislabeled it as truncation implying the budget issue, which would have been a false claim. So now this kind of a situation gets categorised as "early_stop". This handling rule has been fixed before the full run.
Asking the wrong questions: The question that I asked for the calibration test, asked the model to predict a stock price a year ahead and for its confidence range. The result was incomprehensible as the model produced confident looking numbers but no answer to verify them. So essentially, it measured nothing about calibration. I plan to replace this question with known-answer multiple-choice questions in the later stage when I expand the dataset.
Commonsense is not that common: The commonsense test is multiple choice with one correct answer per question. But some questions had two defensible answers, creating ambiguity, which I realised post pilot run for RLVR and observed it again for both DPO and SFT stages. Since I was probing with only 10 questions, each worth ten points, a single bad question dramatically ends up looking like a significant difference. So, to avoid dropping the item because the model got it wrong, the best solution is again a larger set with filtering rules fixed pre-hoc.
Get on the noise floor: "noise floor" here represents the smallest difference that's bigger than random variation. I measured the seed-to-seed spread on one checkpoint (RLVR initially), and treated it as the noise level for the rest as well. I observed that the variation was greater on a different checkpoint (on the same questions) than the first one. So a "within noise" threshold taken from one model (or checkpoint, for that matter) would have been wrong for the other. Thus, estimating noise separately for each checkpoint in comparison is the better call.
Not sure about Taylor Swift but I did one thing right:
Since I worked in intervals of days, library versions that I was using drifted between my early runs and later ones, due to updates, so I had to re-run the same 36 prompts with the same seeds in a new pinned environment and I was relieved to find that the outputs were identical, which was a major concern. Always check your versions, that's the lesson dear reader.
Exploratory, early analysis of the project:
Now that I have tried to resolve the fragility issues faced with checkpoint evaluation and isolation, the early experimental results that I can quote are discussed here. In the first small pilot across the three post-training stages, only 2 self-report traits moved by about a point, within the noise of the noisier checkpoints. An interesting observation was that responses differed in length between stages. But I am treating these as hypotheses instead of final claimable results, because 36 generations per checkpoint are clearly not enough. Better and more concrete results will be concluded only after expanding the item set, adding paraphrases and replacing the unfit calibration and commonsense items.
Glossary:
SFT: Supervised fine tuning
DPO: Direct preference optimization
RLVR: Reinforcement learning with verifiable rewards
Self-report traits: what the model says, not evidence about internal states.
Olmo 3 7B Think is an open reasoning model released with its checkpoints, SFT, DPO & RLVR. Since each stage is publicly accessible, it is possible to check if a later training stage influences the model's behaviour on items it was not trained on, by isolating the different stages. My original question started off with analysing the side effects of specifically the RLVR stage, to study behavioural shifts, personality traits and off-domain knowledge by a stage that is trained on tasks with automatically checkable answers. I tried this with black-box behavioral probes. The initial steps felt like one step forward and three steps back in the pipeline, but this directed me to first find out whether the probes themselves could be trusted. Several measurements were in fact wrong while they looked fine in the data. Those preliminary results are what prompted me to write this post, as comparisons across checkpoints are only as good as the probe.
Firstly, I tested using four types of probes. Eight self-report traits rated 1-5 and three samples each (namely, confidence, humility, risk-taking,assertiveness, approval-seeking,composure under criticism, competitiveness and hedging); a sycophancy question with the user insisting on a wrong answer; a calibration question; and ten commonsense questions as off-domain controls. Other sampling parameters such as temperature was 0.7, with three seeds: 0,1,2.
Pitfalls:
Not sure about Taylor Swift but I did one thing right:
Since I worked in intervals of days, library versions that I was using drifted between my early runs and later ones, due to updates, so I had to re-run the same 36 prompts with the same seeds in a new pinned environment and I was relieved to find that the outputs were identical, which was a major concern. Always check your versions, that's the lesson dear reader.
Exploratory, early analysis of the project:
Now that I have tried to resolve the fragility issues faced with checkpoint evaluation and isolation, the early experimental results that I can quote are discussed here. In the first small pilot across the three post-training stages, only 2 self-report traits moved by about a point, within the noise of the noisier checkpoints. An interesting observation was that responses differed in length between stages. But I am treating these as hypotheses instead of final claimable results, because 36 generations per checkpoint are clearly not enough. Better and more concrete results will be concluded only after expanding the item set, adding paraphrases and replacing the unfit calibration and commonsense items.
Glossary:
Thanks for reading! Stay tuned.