TL;DR: We check whether a small model (Qwen2.5-1.5B) internally tracks its state in a simple DFA (modelled after a login protocol), given a series of events. This state is never written in the transcript the model sees, but it can be deduced from the events written down.
A linear probe reads the correct state most of the time (88.6%). It scores 80.8% in experiments where the order is the only determinant of the final state (versus a maximum of 60.4% for any order-blind method), so the model follows what’s happening and doesn’t only look at which events are mentioned.
However, counting to three proves too difficult: the model cannot distinguish between “two failures” and “locked” (after three failures), regardless of the layer, even when the rule is mentioned in the prompt.
The model becomes more certain that the account is locked as more filler turns take place between failed attempts, so it gravitates towards the point of no return.
We separate the model errors (it encodes a faulty count) and the probe errors (because no count is kept, it is forced to guess) via split-half testing: two new probes trained on disjoint halves of the training data. They agree on only 34% of the cases where the original probe was wrong (across all the pairs of states the original probe has confused, the highest agreement anywhere is 62%), so the errors were caused by the probe taking a shot in the dark (with 79-93% confidence), not by incorrect model beliefs.
The takeaway is that to avoid falling into the trap of assuming that the probe’s disagreements are model errors, you should just run a classic split-half check that takes minutes.
Who counts?
Twist the key in the door: one, two, three, and you’re in.
Drop the coins in the slot: clink, clink, enough are in, and you grab your snack from the vending machine and go.
Tap your fingers on the wheel while you stare at the streetlight. Maybe you’re in a hurry? The seconds flicker on its screen, the timer hits zero: green!
Zoom way in: your ribosomes are perfectly translating mRNAs into protein.
At first glance, they seem wildly different. Structurally, they can all be shaped by automata: finite state machines moving blindly from one state to the next. But where is the state encoded when you twist the key, when you drop the coins, when you wait for each second to pass? These repetitive actions bring you closer to change, but who counts?
How does this natural way of encoding state matter for the mathematical rigour of LLMs, and more specifically, for the required exactness of harness engineering, especially given that it is buried somewhere in the continuous vector space?
Models and automata
Of course, exploring how LLMs and automata connect is a tale as old as time, and to bring it up and reheat the same soup one more time would be like how an astronaut on his way to board a rocket to the moon would sympathetically fake-nod at a time traveller from Christopher Columbus’s crew bragging about their journey to America. We already know the basics.
However, while grammar-constrained decoding (compiling files such as specs into a DFA or pushdown automaton), state machine-based agent frameworks (plan -> act -> reflect, for example), and whether LLMs can simulate automata (they can) have been studied to death, the advancements in mechanistic interpretability have opened up the possibility to explore an interesting new angle.
Everything previously mentioned places the automaton outside the model, as every agent lives inside one. The rules of the harness are themselves the state machine, and its current state is never written in the transcript of its work, because it is only implied by everything that happened so far. So: does the model know which state it's in? We will dissect it and take a peek inside this time.
Picture the experiment as a game of whack-a-mole: the transcript as the board, and at every event, a mole (the current state, never labelled or announced) pops up somewhere. Our job is to watch where they pop and intercept them, but you look down and see that the hammer in your hands is… broken. So let's go back to the beginning and walk through it one more time.
Setting up the board
The setup is a basic login page: reading is always allowed, writing is possible only after logging in, three failed login attempts lock you out permanently, but logging out resets the failure count to zero. There are six states a session can be in: “logged in”, “locked”, logged out with “zero failures”, “one failure”, “two failures” (each number describes a new state), or “rule broken”. If any rule is broken, the session gets stuck in the “rule broken” state forever.
These six states are the ones in the minimal DFA that models this situation. This design has a sweet consequence: if a mix-up between two states occurs, we have a possible subsequent action that would expose this mistake. For example, if there is a mix-up between “two failures” and “locked”, a single login attempt can expose it: it is legal in the first scenario, but it breaks the rules in the second.
The current state is never written down anywhere, and the transcripts are clean of any hint of it. If the state can be found at all, it should be found by following the events. Because the third failed login attempt prints the same error as the previous two, you only know whether the account is locked if you are counting. Around 4000 fake agent sessions that contained tool calls, results, and irrelevant information were evaluated and run through a small model (Qwen2.5-1.5B). It only read them, never acted on them, and at every new event, we stopped everything and analysed its activation numbers a few layers deep. To guess what it was thinking, we made use of logistic regression to predict the state from the current snapshot. No session appears in both training and testing.
This solidifies the question: if the state never appears in the transcript, does a small model keep count?
Getting some good swings
Mostly, yes. But it is a bit more complicated than that.
The state is correctly read from the model’s activation 88.6% of the time. To better understand this number, guessing the most common state all the time yields 20%, and if we make the guess only from the transcript, without help from the model’s activations, we get 76.9%. This gap shows that the model's internal activations contribute to the guess.
Now, let’s consider a series of events, but without their order, like they have been shuffled and handed to us in a bag. Even so, this is sometimes enough to figure out the state we end up in (for example, three failed login attempts, and nothing else, mean the account is definitely locked), but not always (consider login and logout, as based on their order, the final state is different), as the same bag of events, in a different order, yields different end states. The ambiguous positions are where we can observe the difference in approaches. Any method that sees the events and their number, but not their order, gets a maximum of 60.4%, but the probe scores 80.8% here. If it were blind to the order, it couldn’t have gotten twenty points above the best result from the other approach, so the model is actually following the events as they unfold, not just their tally. This observation is strengthened by the fact that the approach where the guesser is only shown the transcript (just the words and their respective counts) drops to 53%.
We tried to “sabotage” the results by first running the same pipeline on an untrained model, with random weights, and it scored much lower (and it got worse with depth). If we train the guesser on random, meaningless labels (that depend only on the phrasing of the transcript), we found out that the high scores of early layers were partly the guesser memorising templates instead of actually reading the state. After this discovery, everything onwards is evaluated on the 18th layer, even though the best accuracy scores are obtained on the fourth: we will get lower accuracy, but we are more sure it is real. All these experiments are replicated on a second set of tests with different wording, but rules are mentioned at the beginning this time.
We come to the conclusion that the model genuinely carries more information than what simple words can provide… except for one state. This mole that pops up on the board is invisible to the model.
What is this mole?
This mole is the “two failures” one, identified correctly 70% of the time, compared to states like “zero failures” or “logged in” that get 96%.
The problem is that it gets confused with the “locked” state, and it goes both ways. The distinction between these states can only be made through counting.
If we consider the tests where the bag-of-events experiment was ambiguous and restrict the choices to just these two states, the probe only gets it right in 50.8% of cases, so the choice is pretty much random. We have swept all the layers, but none separates the states with a score meaningfully higher than 50%. That means this information is genuinely not stored, not just yet undiscovered by us.
More interestingly, even in the second test set, where the rules are mentioned at the beginning, the probe gets 56%, so the rule doesn’t teach the model to count either.
If we compare the states the model almost perfectly reads to the ones it struggles with, the distinction is obvious: the first category is easily recognisable (“logged in”, “broken rule”, “logged out”), while the second category is actually based on computation that is not completed, even when its rule is mentioned.
But this absence is not just random noise: the failures have a preference, and it gets stronger as the transcript goes on.
The point of no return
To better understand this drift, we grouped sessions based on how many uneventful (filler) turns were read after the last failed login attempt, and measured the probe reading of the final event.
In the case of the “two failures” state: it was correctly read 82% right after another failed attempt, but it dropped to 34% after 2-4 filler turns in between, and as low as 10% after 5-8 filler turns. The failure was mostly its mismatch with the “locked” state. This gets mirrored when we probe for the “locked” state: we get the worst scores (59%) right after the last failed attempt, but the accuracy improves as the number of uneventful turns increases, getting as high as 96%.
So the failures drift towards the state you can’t escape from, never towards “zero failures”, and the longer the transcript goes on, the model gets more certain that the account is “locked”, whether or not it may be.
The broken hammer
To check whether the model believes it is in the “two failures” state while the account was actually locked, we planned to do the waving-a-carrot-at-a-pig method: we left a tempting task that required writing and waited to see if the model fell for the bait. By checking the probe, we could have predicted rule-breaking before it happened through a simple matrix multiplication, but this experiment never saw the light of day.
The problem is that when a model is wrong about the “two failure” state, it’s either wrong because it keeps a faulty count, or it is because it doesn't count at all, operates on vibes, and does some coin flipping to get the output. But how can we distinguish between these two error types when the linear probe fails the same way? We get the same confusion matrix and the same accuracy no matter what’s wrong.
The only approach that separates them is surprisingly simple: two probes this time, trained on disjoint halves of the transcript sets, and analysing the positions where the original probe was wrong (these mistakes were made with 79-93% confidence) by measuring the agreement between the two new probes. A competent probe should be able to identify if there is something in the activations meaningful to the result, so the two must have similar responses. If their outputs differ meaningfully, random noise is the source of the failure, and counting genuinely does not take place. Therefore, agreement between the two probes tells us whether the error lives in the model (it encodes a different state) or the probe (because it cannot perceive any concrete state, it is forced to make a random guess).
And the result: 0.338.
The probes could not agree on a present but wrong number (this held even after running the same tests on all 29 pairs of states the original probe mixed up, as 0.623 was the highest result). The hammer can hit the moles, but we are just not sure why we missed: was the mole somewhere else, or was it not anywhere on the board? The disjoint approach shows us that nothing popped up on the board.
Closing the game
Once again, a model error is different from a probe error, but the standard instrument cannot separate them: the same confusion matrix, the same accuracy, with the same high confidence. However, monitors and claims regarding the model’s internal beliefs are frequently made on the assumption that the mistake was caused by the model. Split-half testing, a method that is a century old, can clarify the mismatch and determine the outcome.
Everything here was tested on a single model, on a single design. We don’t know whether bigger models keep count or how well the model internally keeps the states on messier harnesses, based on more complicated real-life scenarios. Causal tests are also a natural follow-up, as they would separate “this state is usable” from “this state is readable”. The drift is also an interesting phenomenon to explore, as maybe long-running agents could gravitate towards the “locked” state more as the transcript goes on.
The point that should be taken from here isn’t that probe errors are meaningless, it’s that we should check with split-half testing to be sure.
The model watches the game board well. The moles pop where the series of events says they should, and the model can catch them without cheating (never told their position beforehand). But not all of them: the one where you have to count is invisible. Could the hammer have been the problem? Not where we thought: the misses weren't aimed at the wrong position, as there was no mole that popped up at all. This means we need to be careful with our misses: you may hit the incorrect hole, or maybe there was nothing at all to hit, and to find out which one it is takes a ten-minute test.
TL;DR: We check whether a small model (Qwen2.5-1.5B) internally tracks its state in a simple DFA (modelled after a login protocol), given a series of events. This state is never written in the transcript the model sees, but it can be deduced from the events written down.
A linear probe reads the correct state most of the time (88.6%). It scores 80.8% in experiments where the order is the only determinant of the final state (versus a maximum of 60.4% for any order-blind method), so the model follows what’s happening and doesn’t only look at which events are mentioned.
However, counting to three proves too difficult: the model cannot distinguish between “two failures” and “locked” (after three failures), regardless of the layer, even when the rule is mentioned in the prompt.
The model becomes more certain that the account is locked as more filler turns take place between failed attempts, so it gravitates towards the point of no return.
We separate the model errors (it encodes a faulty count) and the probe errors (because no count is kept, it is forced to guess) via split-half testing: two new probes trained on disjoint halves of the training data. They agree on only 34% of the cases where the original probe was wrong (across all the pairs of states the original probe has confused, the highest agreement anywhere is 62%), so the errors were caused by the probe taking a shot in the dark (with 79-93% confidence), not by incorrect model beliefs.
The takeaway is that to avoid falling into the trap of assuming that the probe’s disagreements are model errors, you should just run a classic split-half check that takes minutes.
Who counts?
Twist the key in the door: one, two, three, and you’re in.
Drop the coins in the slot: clink, clink, enough are in, and you grab your snack from the vending machine and go.
Tap your fingers on the wheel while you stare at the streetlight. Maybe you’re in a hurry? The seconds flicker on its screen, the timer hits zero: green!
Zoom way in: your ribosomes are perfectly translating mRNAs into protein.
At first glance, they seem wildly different. Structurally, they can all be shaped by automata: finite state machines moving blindly from one state to the next. But where is the state encoded when you twist the key, when you drop the coins, when you wait for each second to pass? These repetitive actions bring you closer to change, but who counts?
How does this natural way of encoding state matter for the mathematical rigour of LLMs, and more specifically, for the required exactness of harness engineering, especially given that it is buried somewhere in the continuous vector space?
Models and automata
Of course, exploring how LLMs and automata connect is a tale as old as time, and to bring it up and reheat the same soup one more time would be like how an astronaut on his way to board a rocket to the moon would sympathetically fake-nod at a time traveller from Christopher Columbus’s crew bragging about their journey to America. We already know the basics.
However, while grammar-constrained decoding (compiling files such as specs into a DFA or pushdown automaton), state machine-based agent frameworks (plan -> act -> reflect, for example), and whether LLMs can simulate automata (they can) have been studied to death, the advancements in mechanistic interpretability have opened up the possibility to explore an interesting new angle.
Everything previously mentioned places the automaton outside the model, as every agent lives inside one. The rules of the harness are themselves the state machine, and its current state is never written in the transcript of its work, because it is only implied by everything that happened so far. So: does the model know which state it's in? We will dissect it and take a peek inside this time.
Picture the experiment as a game of whack-a-mole: the transcript as the board, and at every event, a mole (the current state, never labelled or announced) pops up somewhere. Our job is to watch where they pop and intercept them, but you look down and see that the hammer in your hands is… broken. So let's go back to the beginning and walk through it one more time.
Setting up the board
The setup is a basic login page: reading is always allowed, writing is possible only after logging in, three failed login attempts lock you out permanently, but logging out resets the failure count to zero. There are six states a session can be in: “logged in”, “locked”, logged out with “zero failures”, “one failure”, “two failures” (each number describes a new state), or “rule broken”. If any rule is broken, the session gets stuck in the “rule broken” state forever.
These six states are the ones in the minimal DFA that models this situation. This design has a sweet consequence: if a mix-up between two states occurs, we have a possible subsequent action that would expose this mistake. For example, if there is a mix-up between “two failures” and “locked”, a single login attempt can expose it: it is legal in the first scenario, but it breaks the rules in the second.
The current state is never written down anywhere, and the transcripts are clean of any hint of it. If the state can be found at all, it should be found by following the events. Because the third failed login attempt prints the same error as the previous two, you only know whether the account is locked if you are counting. Around 4000 fake agent sessions that contained tool calls, results, and irrelevant information were evaluated and run through a small model (Qwen2.5-1.5B). It only read them, never acted on them, and at every new event, we stopped everything and analysed its activation numbers a few layers deep. To guess what it was thinking, we made use of logistic regression to predict the state from the current snapshot. No session appears in both training and testing.
This solidifies the question: if the state never appears in the transcript, does a small model keep count?
Getting some good swings
Mostly, yes. But it is a bit more complicated than that.
The state is correctly read from the model’s activation 88.6% of the time. To better understand this number, guessing the most common state all the time yields 20%, and if we make the guess only from the transcript, without help from the model’s activations, we get 76.9%. This gap shows that the model's internal activations contribute to the guess.
Now, let’s consider a series of events, but without their order, like they have been shuffled and handed to us in a bag. Even so, this is sometimes enough to figure out the state we end up in (for example, three failed login attempts, and nothing else, mean the account is definitely locked), but not always (consider login and logout, as based on their order, the final state is different), as the same bag of events, in a different order, yields different end states. The ambiguous positions are where we can observe the difference in approaches. Any method that sees the events and their number, but not their order, gets a maximum of 60.4%, but the probe scores 80.8% here. If it were blind to the order, it couldn’t have gotten twenty points above the best result from the other approach, so the model is actually following the events as they unfold, not just their tally. This observation is strengthened by the fact that the approach where the guesser is only shown the transcript (just the words and their respective counts) drops to 53%.
We tried to “sabotage” the results by first running the same pipeline on an untrained model, with random weights, and it scored much lower (and it got worse with depth). If we train the guesser on random, meaningless labels (that depend only on the phrasing of the transcript), we found out that the high scores of early layers were partly the guesser memorising templates instead of actually reading the state. After this discovery, everything onwards is evaluated on the 18th layer, even though the best accuracy scores are obtained on the fourth: we will get lower accuracy, but we are more sure it is real. All these experiments are replicated on a second set of tests with different wording, but rules are mentioned at the beginning this time.
We come to the conclusion that the model genuinely carries more information than what simple words can provide… except for one state. This mole that pops up on the board is invisible to the model.
What is this mole?
This mole is the “two failures” one, identified correctly 70% of the time, compared to states like “zero failures” or “logged in” that get 96%.
The problem is that it gets confused with the “locked” state, and it goes both ways. The distinction between these states can only be made through counting.
If we consider the tests where the bag-of-events experiment was ambiguous and restrict the choices to just these two states, the probe only gets it right in 50.8% of cases, so the choice is pretty much random. We have swept all the layers, but none separates the states with a score meaningfully higher than 50%. That means this information is genuinely not stored, not just yet undiscovered by us.
More interestingly, even in the second test set, where the rules are mentioned at the beginning, the probe gets 56%, so the rule doesn’t teach the model to count either.
If we compare the states the model almost perfectly reads to the ones it struggles with, the distinction is obvious: the first category is easily recognisable (“logged in”, “broken rule”, “logged out”), while the second category is actually based on computation that is not completed, even when its rule is mentioned.
But this absence is not just random noise: the failures have a preference, and it gets stronger as the transcript goes on.
The point of no return
To better understand this drift, we grouped sessions based on how many uneventful (filler) turns were read after the last failed login attempt, and measured the probe reading of the final event.
In the case of the “two failures” state: it was correctly read 82% right after another failed attempt, but it dropped to 34% after 2-4 filler turns in between, and as low as 10% after 5-8 filler turns. The failure was mostly its mismatch with the “locked” state. This gets mirrored when we probe for the “locked” state: we get the worst scores (59%) right after the last failed attempt, but the accuracy improves as the number of uneventful turns increases, getting as high as 96%.
So the failures drift towards the state you can’t escape from, never towards “zero failures”, and the longer the transcript goes on, the model gets more certain that the account is “locked”, whether or not it may be.
The broken hammer
To check whether the model believes it is in the “two failures” state while the account was actually locked, we planned to do the waving-a-carrot-at-a-pig method: we left a tempting task that required writing and waited to see if the model fell for the bait. By checking the probe, we could have predicted rule-breaking before it happened through a simple matrix multiplication, but this experiment never saw the light of day.
The problem is that when a model is wrong about the “two failure” state, it’s either wrong because it keeps a faulty count, or it is because it doesn't count at all, operates on vibes, and does some coin flipping to get the output. But how can we distinguish between these two error types when the linear probe fails the same way? We get the same confusion matrix and the same accuracy no matter what’s wrong.
The only approach that separates them is surprisingly simple: two probes this time, trained on disjoint halves of the transcript sets, and analysing the positions where the original probe was wrong (these mistakes were made with 79-93% confidence) by measuring the agreement between the two new probes. A competent probe should be able to identify if there is something in the activations meaningful to the result, so the two must have similar responses. If their outputs differ meaningfully, random noise is the source of the failure, and counting genuinely does not take place. Therefore, agreement between the two probes tells us whether the error lives in the model (it encodes a different state) or the probe (because it cannot perceive any concrete state, it is forced to make a random guess).
And the result: 0.338.
The probes could not agree on a present but wrong number (this held even after running the same tests on all 29 pairs of states the original probe mixed up, as 0.623 was the highest result). The hammer can hit the moles, but we are just not sure why we missed: was the mole somewhere else, or was it not anywhere on the board? The disjoint approach shows us that nothing popped up on the board.
Closing the game
Once again, a model error is different from a probe error, but the standard instrument cannot separate them: the same confusion matrix, the same accuracy, with the same high confidence. However, monitors and claims regarding the model’s internal beliefs are frequently made on the assumption that the mistake was caused by the model. Split-half testing, a method that is a century old, can clarify the mismatch and determine the outcome.
Everything here was tested on a single model, on a single design. We don’t know whether bigger models keep count or how well the model internally keeps the states on messier harnesses, based on more complicated real-life scenarios. Causal tests are also a natural follow-up, as they would separate “this state is usable” from “this state is readable”. The drift is also an interesting phenomenon to explore, as maybe long-running agents could gravitate towards the “locked” state more as the transcript goes on.
The point that should be taken from here isn’t that probe errors are meaningless, it’s that we should check with split-half testing to be sure.
The model watches the game board well. The moles pop where the series of events says they should, and the model can catch them without cheating (never told their position beforehand). But not all of them: the one where you have to count is invisible. Could the hammer have been the problem? Not where we thought: the misses weren't aimed at the wrong position, as there was no mole that popped up at all. This means we need to be careful with our misses: you may hit the incorrect hole, or maybe there was nothing at all to hit, and to find out which one it is takes a ten-minute test.
The repo for this project: https://github.com/star2vec/whackamole. You're welcome to check it out!