Jeff Schecter

"Humans are often stupid in that way too"—we've said it of bots from GOFAI through LLMs + RLVR; it's always been true but not predictive or productive. I think the temptation arises because we recognize the same failure modes in ourselves and fellow meatbags. But whatever the pathology du jour may be, in us it's never quite so pathological; whereas to the bots it's crippling, the glass that the ceiling is made of.

These pathologies arise for specific reasons that point to real deficits. For each bot generation going back lo unto the dawn of time, you can tr... (read more)

Every OpenAI tokenizer I can find splits the sting "colorado" into two tokens, "color" "ado". More recent tokenizers split "tornado" into "t" "ornado".... but the GPT3 tokenizer instead yields "torn" "ado".

To me, this is obviously not a case of the state of Colorado being associated semantically with software engineering because it's a big industry in Denver and Boulder. It's a bigram unifier that responds to ado. It's a V-shape detector. Or it was at some early point in the training process.

From there, it muscled into things semantically adjacent to Color... (read more)

That's true! OpenAI's report notes however that before the incident, agents had already started using Artifactory message boards in training from May 8 onward. Reinforcement there may have been one factor leading so many agents to do so again during evals.

They imply but do not state explicitly that some training rollouts involving the Artifactory message board received reward, and that frequency of board use increased throughout training. They mention a single touch of Artifactory on May 8, the next on May 12, several more May 13, and "Sustained high-volum... (read more)

Better sandboxing has a very practical and straightforward effect on prosaic alignment!

Leaky sandboxes cause more rewards for cheating during RLHF.

Rewards for egregious behavior during training cause egregious behavior in deployment.

If you look at say https://arxiv.org/pdf/2502.05171 Figure 8, it does not appear that increasing recurrent depth in deployment is a useful "test-time scaling" axis: which is to say, when trained at a max depth of 32, setting depth to 64 at inference time bought them no improvement.

Thought who knows how RL would change the story, or if their model in practice has similar characteristics to the research products visible from the last few years.

From what I understand that's basically correct; you go to depth recurrence when you're bottlenecked by memory or training data but still want to increase compute.

With the memory situation being what it is, I can see how a bit of depth recurrence might become attractive.

From what I understand the world is not shaped in a way that makes "scaling latent reasoning unto infinity while having gotten rid of visible CoTs entirely" appealing or likely.

Which is to say, spending your marginal flops on recurrent depth seems to be Pareto-dominated by all of: more pretraining tokens, more RL, and more params—unless you find yourself faced with some rather extreme constraints. Like "I'm out of VRAM, but my MFU is low, I can't get my hands on any more pretraining data, I've already run several epochs on the data I do have, I'm hit sever... (read more)