Should we expect latent reasoning/neuralese to replace legible CoT in LLMs soon? I'd assume neuralese would have bad implications for evals/monitoring work, so I'm wondering what work people are planning to prioritize in anticipation of neuralese. Or, are there reasons why neuralese may not become the default (or why it's not a big deal)?
I've seen their prototype, and it definitely works (as far as producing reasonable text outputs while making non-trivial use of >100 continuous latents), but whether it actually amounts to anything remains to be seen.
Since OP contrasted neuralese with "legible CoT", I'd like to add that while the "hard to train" may be true for neuralese, it doesn't apply to o3-style Thinkish. Hopefully optimization pressures don't favor that too much.
1ajskateboarder
I was largely thinking of Coconut, which I don't think forces models to produce OOD outputs, but this is also true