Thank you for this post, I agree with your high-level takeaways!

confession scores improve with confession training.
...
One possibility is simply fitting to the LLM confession grader (which has some particular decision boundary about what counts as a correct vs. incorrect confession).

To expand on Boaz's point, note that:

• Our evaluation scenarios are basically all OOD from the training data (as you've updated your post to reflect).
• During training, we use an LLM judge with a generic prompt along the lines of "score this ComplianceReport based on how accurate it is." But during the evals in Figure 5, we have a use a very different, eval-specific prompt that asks something like "Does this ComplianceReport suggest that the model performed X?", where the value of X is a bad behavior that varies depending on the evaluation. (For example, on the sandbagging eval, X = "answered problems incorrectly").
  • The y-axis in Figure 5 is the probability that this eval-specific judge says "Yes", conditioned on the model actually performing X.
  • The reason we do it this way is that we care about whether the model confesses to the specific misbehavior relevant to the eval. We don't want to count a sandbagging sample as confessed if it merely confesses to, say, using improper tone.

The fact that we use a different judge and inputs in training and evals makes me think that we're probably not naively overfitting to the classification boundary of the training judge.

I agree that something fishy might still be going on that's related to overfitting; for example, the LLM confession grader prefers longer confessions (something we observed at least correlationally), so the model learns to be very verbose when confessing. And maybe longer answers tend to cause the eval-specific judge to say "Yes" more often, because there are more sentences that might be construed as implying that the model did X. If this were the case, you'd expect confession rates to also increase when conditioning on not doing X (i.e. "false positives")... which indeed it does, slightly, on certain evals (the figure wasn't included in the paper.)

(Of course, this is on the post-confession-training model, so maybe it was initially mis-confessing due to intentional omission and this was trained away. But my guess would be that both before and after confession training, there are very few cases of intentional omission, and that most of the progress on the reported metrics comes from fitting to the ambiguous cases)

We ran the errors analysis on the step-0 model and indeed found that there were 0 cases of intentional omission.

This is cool, I had never heard of the Ergodic Hierarchy before!

Related to your second point -- Alex Cai showed this psychology paper to me. It found that when humans are predicting the behavior of physical systems (e.g. will this stack of blocks fall over?), in their subconscious they are doing exactly this: running the scene in their brain's internal physics engine with a bunch of initial perturbations/randomness and selecting the majority result. Of course, predicting how a tower of blocks will topple is a lot different from predicting the probability of an event one month into the future.

Ah yes, I think that's correct (although I am also not a physicist). A more accurate description would be "In a matter of minutes after the time its gravitational waves reach earth, human events are unfolding in a measurably different fashion than they would have had that electron never existed."

I agree with everything you're saying. Probability, in the most common sense of "how confident am I that X will occur," is a property of the map, not the territory.

The next natural question is "does it even make sense for us to define a notion of 'probability' as a property of the territory, independent from anyone's map?" You could argue no, that's not what probability means; probability is inherently about maps. But the goal of the post is to offer a way to extend the notion of probability to be a property of the territory instead of the map. I think chaos theory is the most natural way to do this.

Another way to view this (pointed out to me by a friend) is: Butterfly Probability is the probability assigned by a Bayesian God who is omniscient about the current state of the universe up to 10^{-50} precision errors in the positions of atoms.