This is a special post for quick takes by sidbaines. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Another reason why looped architectures might be extra-bad: they could lead to stronger abilities to gradient hack during RL (ie, by avoiding exploring trajectories which could cause a model to update in ways it deems 'bad'). If a model is able to, within a single forward pass, realise that it is in an (e.g. alignment) RL env, it could choose to emit a next-token which won't lead it down a trajectory which would cause it to be updated in the model-undesired way.
In some senses this is just a special case of "looped models are bad for CoT monitoring", but this seems like a meaningfully more difficult thing to avoid: even if it is possible to train reliable deployment-time monitors for a final checkpoint model (e.g. based on latents), in order to avoid this ‘hidden gradient hacking’ during RL, monitors would need to be re-trained regularly (which seems unlikely to happen).
Are you arguing that looped architectures are bad, or that neuralese in general is bad for gradient/exploration hacking? I'm confused about why your argument wouldn't apply, for instance, to parallel latent models.
Good point. The argument I was making would hold for any architecture which is trained/designed for high latent reasoning capability (including ‘parallel latent models’ if I am interpreting that correctly). But on ‘neuralese is bad in general’, it’s probably worth clarifying that I see ‘models trained for high latent reasoning capability’ as different (and worse) from ‘regular models which happen to learn neuralese’. First, I think that current neuralese is still somewhat interpretable, so out-of-the-box monitors stand more of a chance at catching gradient hacking in 2026-neuralese than things we can’t see at all. Second, to the extent that neuralese will get harder to understand, I expect that to happen specifically over the course of RL, so we’ll have a chance to i) avoid neuralese and ii) catch early instances of gradient hacking. ‘High latent reasoning models’ on the other hand (may) arrive at RL with the strong latent ability.
That said, I think that if we do get to full-blown neuralese then this would be ~as bad for the gradient hacking concern I raised.