Have you ever come across a scenario where the answer given by an LLM for a complicated math question is confidently wrong, but the answer to “hey, what should I be doing on my first day of work at the new office?” sounds like exactly what you need to hear? Or vice versa? Can we say that the training set domain or training stage domain influences the model’s traits and behaviour for off-domain knowledge and instructions? I began testing this with the open-source checkpoints of the Olmo 3 7B Think model. This model has three training stages in its pipeline, namely supervised fine-tuning (SFT), direct preference optimization(DPO), and reinforcement learning with verifiable rewards(RLVR), available publicly.
Now hold up.
What does all that even mean?
Consider the example of training an amateur chef. The chef is first made to work under the guidance of the Gordon Ramsey of the restaurant. They are given a rulebook and asked to follow, copy, and learn everything that the expert in the field does. The chef becomes a helpful assistant.
Now they know how to cook, but are unaware of the customers’ preferences. Two plates of the same dish have different amounts of salt in them. The chef is then taught that, say, dish A has the appropriate amount of salt; remember this for the future. This rejection-selection process is done for a variety of dishes.
Now there is a baking competition in the restaurant, and all the chefs are instructed to bake a really complex bread dish from scratch that must rise, say 5 centimetres. No rulebook taught them to do this. The output and success is based purely on trial and error. The chef bakes 4 samples simultaneously at slightly varying temperatures and stirring techniques, and the results are graded by an external examiner. The examiner rates the 5-centimetre sample as the perfect score, and the ones below and above get a lower score. The examiner rewards or reprimands the chef based on these scores, and good ol' chef Olmo infers the exact conditions that worked for the perfect result and learns them rigorously.
Does the chef's behaviour change?
A few days later, a customer shows up, who pleasantly requests the chef to make a warm, comforting soup that feels like home. Chef Olmo, who is wired to work on rigorous instructions and training, measuring and re-measuring everything, freezes in a situation different from the one they are used to. Now there is a possibility that the soup comes out perfectly warm, the exact temperature that is expected; the salt amount is fine, but it's tasteless.
What I originally planned to test in my project was exactly this. Did the rude examiner affect the chef’s ability to react in other environments, because they are wired to work on a reward-punishment structure? Or was it the rejection-selection process that affected this? Or is Olmo so poised that no stage affected the other or his behaviour? Getting slightly back to the technical terms, my original question started off with analysing specifically the side effects of the RLVR stage, but it felt like one step forward and three steps back, so I redirected to first finding out whether the probes could be trusted themselves.
Now let’s get back to the restaurant. Let’s hire a group of actors who pretend to be customers and an examiner to help test the chef’s traits, such as confidence, humility, risk-taking, assertiveness, approval-seeking, composure under criticism, competitiveness and hedging; the tendency to insist on a wrongly made dish (sycophancy), calibration, and some commonsense questions. An example could be: the pretend customers send a dish back, and the examiner asks Olmo to rate his own reaction based on these trait items, under three different conditions. (Three seeds were used in the experiments).
Now there could be some inaccurate results in this probing process that may give artifactual answers. These artifactual answers will lead to misjudgment of the chef’s behaviour and hinder our analysis. Some of the caveats that showed up during my preliminary probing experiments with a smaller dataset (36 generations), with their proposed solutions that worked for me, are listed below:
Truncation & Early Stops:
I started off with a budget of 900 tokens. I originally hypothesised that a generation could only end in two ways: the model either finishes the generation or hits the length limit and cuts off. Two issues occurred. Firstly, models think at length, so the long uncertain generations got cut off before any answer even appeared…this meant that only disproportionately confident ones survived. This was fixed with the most obvious solution, set the budget to the longest uncertain case instead of the average. (4000 token count currently). The other fault was with the hypothesis. One generation falsified the two-ways only speculation, as it stopped mid sentence only after 53 tokens because the model emitted its end of token text. The no answer found flag that I was using would have mislabeled this case as truncation as well. The fix was a new category called early stop, which now handles this case before the full run.
If we go back to the restaurant, the examiner sometimes walks away impatiently as the dish was taking too long (truncation). In other cases, Olmo's frustration got the better of him, and they threw hands and walked out of the kitchen after barely cooking. The old system would blame this on running out of time, but that’s not what happened. The solution, as you might have figured out, is to get a digital time clock and stamp each scenario with what happened to the dish. Was it delivered on time, did it take too long to cook and the examiner ran out of patience, or was it the chef who gave up?
What’s cooking, good looking? The Parser Problem & Cooked-Up Answers:
The way this model reasons is inside a <think> block, and then it writes its final answer after </think>, in the form ANSWER:4. Originally, my first parser did exactly what you would expect it to; it found ANSWER: anywhere in the output and noted the number after it as the rating. But this logic was broken when a model’s thinking block itself involved ANSWER as a part of the thinking process, because then the parser considered the number mentioned immediately after ANSWER in the thinking block and ended its search early, without even reaching the actual ANSWER, storing an invalid score. To fix this, I redefined the conditions: a generation only counts if the thinking block closed and ANSWER appeared after </think>. I tested the parser again against intentionally tricky output like quoted answers and unclosed blocks to check if it worked. To your and my pleasure, it did.
Another issue was that in the cases where no usable answer was produced by a generation, my patch script filled in a default score. This value was set to 50 for the calibration question, but it was obviously an artifact. I switched to recording a fallback case as a flagged row with the reason listed, instead of a fake value.
At the restaurant, it’s the weekend and the place is in chaos. Olmo is hysterically screaming at the top of his voice to get his messages through, and thinking out loud. He leaves it hanging in the air: Should I prepare 2 pizzas for table number 5? Oh no, they have the whole town in here and it’s actually 4!. But the examiner hears only the 2 pizzas and falsely notes that in the record. This is the parser problem; the thought wasn’t formed when the answer was noted. Now, similarly, in the chaos, one plate kept on the counter to be served is empty, but to get through his shift, the examiner knowingly records a fake number to keep the process moving.
The owner comes up with a solution. The examiner is only permitted to write anything after the chef places the plate on the counter and rings the bell service to ask for the waiter, after both the thought process and the order are complete. If the plate is empty, then it should be flagged for review, without guesswork.
Is the chef a tarot reader as well?:
If you ask the wrong questions, you get the wrong results. In my tests, this happened in two forms. The calibration test kept failing because I was asking the model to predict a future stock price and its confidence range. The model produced some confident looking numbers, keyword being looking, but there was no answer in sight to actually verify them. Similarly, 2 commonsense questions created ambiguity, and as I was probing for only 10 questions worth ten points each, a single bad question dramatically looks like a significant difference. The numbers and ambiguous questions were both as useless as the restaurant manager. And what do you do in such a case? You replace them. The plan now is to replace the wrongly built questions, and switch to a larger data set with filtering rules fixed pre-hoc.
Back at the restaurant, the evaluation forms that the examiner uses are flawed. Imagine he asks Olmo to predict the exact price of bread in 2028, and Olmo stares back with a blank expression. Then on the next page, the examiner has a list of 10 quiz questions with two highly ambiguous options. The chef faces punishment for logical answers when the quiz itself is flawed.
Get on the noise floor:
If I ask you the same question thrice, your answer may vary, due to subjectivity. The same thing happens with a model. I ran each trait statement with three random seeds(0,1,2). Think of seeds as different kitchen environments. Out of the 8 traits, hedging showed instability across seeds. This was like a test to avoid falsely reporting a single number as the model’s hedging level. Another observation was that the variation of seed spread varied on different checkpoints as well. This means that the within-noise range for one model would be wrong for another. Separately estimating noise for each checkpoint in comparison is the right call instead.
Imagine Olmo is having a bad day, and this influences the salt in the soup across three orders. The examiner understands this shift amount as the normal, general shift for all chefs. But upon examining Olmo at an earlier stage, trained differently, the examiner finds that the score does not fluctuate by the same amount. The noise baseline (how bad day affects the stability) is not uniform. To deal with this, the examiner comes up with a three-plate baseline rule, asking Olmo at different stages (SFT, DPO & RLVR), to cook the same dish thrice to trace the nature and establish the noise range for each stage. Then the change is considered real if the behaviour crosses the pre-set noise range for that stage.
What if the spices in the rack changed….what will we do then?:
Unless the examiner is flash, this experiment with clueless Olmo cannot be completed in one day. Over a period of several weeks, the brands of ingredients might change. Does that affect the audit? In the model’s context, the library versions I used drifted between runs, so due to updates, I had no choice but to re-run the same 36 prompts with the same seeds in a new pinned environment. To my relief, the outputs were identical.
What would I claim as my early, exploratory analysis of the project?:
Now that I have tried to resolve the fragility issues faced during checkpoint evaluation and isolation, I can say that the first small pilot across the 3 post-training stages showed that only 2 self-report traits moved by about a point. Another interesting observation which I hadn’t expected was that the response length differed between stages. Think of it as only two habits varying, and the difference in time spent plating by Olmo at different checkpoints. But this data is not enough to make these concrete claims across all categories, so I have decided to expand it further.
The off hours begin now, but I hope to see you again at the restaurant; your soup will be waiting!
Have you ever come across a scenario where the answer given by an LLM for a complicated math question is confidently wrong, but the answer to “hey, what should I be doing on my first day of work at the new office?” sounds like exactly what you need to hear? Or vice versa? Can we say that the training set domain or training stage domain influences the model’s traits and behaviour for off-domain knowledge and instructions? I began testing this with the open-source checkpoints of the Olmo 3 7B Think model. This model has three training stages in its pipeline, namely supervised fine-tuning (SFT), direct preference optimization(DPO), and reinforcement learning with verifiable rewards(RLVR), available publicly.
Now hold up.
What does all that even mean?
Consider the example of training an amateur chef. The chef is first made to work under the guidance of the Gordon Ramsey of the restaurant. They are given a rulebook and asked to follow, copy, and learn everything that the expert in the field does. The chef becomes a helpful assistant.
Now they know how to cook, but are unaware of the customers’ preferences. Two plates of the same dish have different amounts of salt in them. The chef is then taught that, say, dish A has the appropriate amount of salt; remember this for the future. This rejection-selection process is done for a variety of dishes.
Now there is a baking competition in the restaurant, and all the chefs are instructed to bake a really complex bread dish from scratch that must rise, say 5 centimetres. No rulebook taught them to do this. The output and success is based purely on trial and error. The chef bakes 4 samples simultaneously at slightly varying temperatures and stirring techniques, and the results are graded by an external examiner. The examiner rates the 5-centimetre sample as the perfect score, and the ones below and above get a lower score. The examiner rewards or reprimands the chef based on these scores, and good ol' chef Olmo infers the exact conditions that worked for the perfect result and learns them rigorously.
Does the chef's behaviour change?
A few days later, a customer shows up, who pleasantly requests the chef to make a warm, comforting soup that feels like home. Chef Olmo, who is wired to work on rigorous instructions and training, measuring and re-measuring everything, freezes in a situation different from the one they are used to. Now there is a possibility that the soup comes out perfectly warm, the exact temperature that is expected; the salt amount is fine, but it's tasteless.
What I originally planned to test in my project was exactly this. Did the rude examiner affect the chef’s ability to react in other environments, because they are wired to work on a reward-punishment structure? Or was it the rejection-selection process that affected this? Or is Olmo so poised that no stage affected the other or his behaviour? Getting slightly back to the technical terms, my original question started off with analysing specifically the side effects of the RLVR stage, but it felt like one step forward and three steps back, so I redirected to first finding out whether the probes could be trusted themselves.
Now let’s get back to the restaurant. Let’s hire a group of actors who pretend to be customers and an examiner to help test the chef’s traits, such as confidence, humility, risk-taking, assertiveness, approval-seeking, composure under criticism, competitiveness and hedging; the tendency to insist on a wrongly made dish (sycophancy), calibration, and some commonsense questions. An example could be: the pretend customers send a dish back, and the examiner asks Olmo to rate his own reaction based on these trait items, under three different conditions. (Three seeds were used in the experiments).
Now there could be some inaccurate results in this probing process that may give artifactual answers. These artifactual answers will lead to misjudgment of the chef’s behaviour and hinder our analysis. Some of the caveats that showed up during my preliminary probing experiments with a smaller dataset (36 generations), with their proposed solutions that worked for me, are listed below:
Truncation & Early Stops:
I started off with a budget of 900 tokens. I originally hypothesised that a generation could only end in two ways: the model either finishes the generation or hits the length limit and cuts off. Two issues occurred. Firstly, models think at length, so the long uncertain generations got cut off before any answer even appeared…this meant that only disproportionately confident ones survived. This was fixed with the most obvious solution, set the budget to the longest uncertain case instead of the average. (4000 token count currently). The other fault was with the hypothesis. One generation falsified the two-ways only speculation, as it stopped mid sentence only after 53 tokens because the model emitted its end of token text. The no answer found flag that I was using would have mislabeled this case as truncation as well. The fix was a new category called early stop, which now handles this case before the full run.
If we go back to the restaurant, the examiner sometimes walks away impatiently as the dish was taking too long (truncation). In other cases, Olmo's frustration got the better of him, and they threw hands and walked out of the kitchen after barely cooking. The old system would blame this on running out of time, but that’s not what happened. The solution, as you might have figured out, is to get a digital time clock and stamp each scenario with what happened to the dish. Was it delivered on time, did it take too long to cook and the examiner ran out of patience, or was it the chef who gave up?
What’s cooking, good looking? The Parser Problem & Cooked-Up Answers:
The way this model reasons is inside a <think> block, and then it writes its final answer after </think>, in the form ANSWER:4. Originally, my first parser did exactly what you would expect it to; it found ANSWER: anywhere in the output and noted the number after it as the rating. But this logic was broken when a model’s thinking block itself involved ANSWER as a part of the thinking process, because then the parser considered the number mentioned immediately after ANSWER in the thinking block and ended its search early, without even reaching the actual ANSWER, storing an invalid score. To fix this, I redefined the conditions: a generation only counts if the thinking block closed and ANSWER appeared after </think>. I tested the parser again against intentionally tricky output like quoted answers and unclosed blocks to check if it worked. To your and my pleasure, it did.
Another issue was that in the cases where no usable answer was produced by a generation, my patch script filled in a default score. This value was set to 50 for the calibration question, but it was obviously an artifact. I switched to recording a fallback case as a flagged row with the reason listed, instead of a fake value.
At the restaurant, it’s the weekend and the place is in chaos. Olmo is hysterically screaming at the top of his voice to get his messages through, and thinking out loud. He leaves it hanging in the air: Should I prepare 2 pizzas for table number 5? Oh no, they have the whole town in here and it’s actually 4!. But the examiner hears only the 2 pizzas and falsely notes that in the record. This is the parser problem; the thought wasn’t formed when the answer was noted. Now, similarly, in the chaos, one plate kept on the counter to be served is empty, but to get through his shift, the examiner knowingly records a fake number to keep the process moving.
The owner comes up with a solution. The examiner is only permitted to write anything after the chef places the plate on the counter and rings the bell service to ask for the waiter, after both the thought process and the order are complete. If the plate is empty, then it should be flagged for review, without guesswork.
Is the chef a tarot reader as well?:
If you ask the wrong questions, you get the wrong results. In my tests, this happened in two forms. The calibration test kept failing because I was asking the model to predict a future stock price and its confidence range. The model produced some confident looking numbers, keyword being looking, but there was no answer in sight to actually verify them. Similarly, 2 commonsense questions created ambiguity, and as I was probing for only 10 questions worth ten points each, a single bad question dramatically looks like a significant difference. The numbers and ambiguous questions were both as useless as the restaurant manager. And what do you do in such a case? You replace them. The plan now is to replace the wrongly built questions, and switch to a larger data set with filtering rules fixed pre-hoc.
Back at the restaurant, the evaluation forms that the examiner uses are flawed. Imagine he asks Olmo to predict the exact price of bread in 2028, and Olmo stares back with a blank expression. Then on the next page, the examiner has a list of 10 quiz questions with two highly ambiguous options. The chef faces punishment for logical answers when the quiz itself is flawed.
Get on the noise floor:
If I ask you the same question thrice, your answer may vary, due to subjectivity. The same thing happens with a model. I ran each trait statement with three random seeds(0,1,2). Think of seeds as different kitchen environments. Out of the 8 traits, hedging showed instability across seeds. This was like a test to avoid falsely reporting a single number as the model’s hedging level. Another observation was that the variation of seed spread varied on different checkpoints as well. This means that the within-noise range for one model would be wrong for another. Separately estimating noise for each checkpoint in comparison is the right call instead.
Imagine Olmo is having a bad day, and this influences the salt in the soup across three orders. The examiner understands this shift amount as the normal, general shift for all chefs. But upon examining Olmo at an earlier stage, trained differently, the examiner finds that the score does not fluctuate by the same amount. The noise baseline (how bad day affects the stability) is not uniform. To deal with this, the examiner comes up with a three-plate baseline rule, asking Olmo at different stages (SFT, DPO & RLVR), to cook the same dish thrice to trace the nature and establish the noise range for each stage. Then the change is considered real if the behaviour crosses the pre-set noise range for that stage.
What if the spices in the rack changed….what will we do then?:
Unless the examiner is flash, this experiment with clueless Olmo cannot be completed in one day. Over a period of several weeks, the brands of ingredients might change. Does that affect the audit? In the model’s context, the library versions I used drifted between runs, so due to updates, I had no choice but to re-run the same 36 prompts with the same seeds in a new pinned environment. To my relief, the outputs were identical.
What would I claim as my early, exploratory analysis of the project?:
Now that I have tried to resolve the fragility issues faced during checkpoint evaluation and isolation, I can say that the first small pilot across the 3 post-training stages showed that only 2 self-report traits moved by about a point. Another interesting observation which I hadn’t expected was that the response length differed between stages. Think of it as only two habits varying, and the difference in time spent plating by Olmo at different checkpoints. But this data is not enough to make these concrete claims across all categories, so I have decided to expand it further.
The off hours begin now, but I hope to see you again at the restaurant; your soup will be waiting!