This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The latest of these (huginn) introduces recurrent latent reasoning, showing signs of life with (possibly unbounded) amounts of compute in the forward pass. Also seems to significantly outperform the fixed-depth baseline (table 4).
Imagine a language model that can do a possibly unbounded amount of internal computation in order to compute its answer. Seems like interpretability will be very difficult. This is worrying because externalised reasoning seems upstream of many other agendas
How can we study these models?
A good proxy right now may be language models provided with hidden scratchpads.
Other kinds of model organism seem really important
If black box techniques don't work well we might need to hail mary on mech interp.
The recurrent paper is actually scary, but some of the stuff there are actually questionable. is 8 layers enough for a 3.5b model? qwen 0.5b has 24 layers.there is also almost no difference between 180b vs 800b model, when r=1(table 4). is this just a case of overcoming insufficient number of layers here?
almost no difference between 180b vs 800b model, when r=1(table 4)
It's a 3B parameter model, so training it for 180B tokens already overtrains it maybe 3x, and training for 800B tokens overtrains it 13x. The loss of compute efficiency from the latter is about 1.6x more than from the former, with 4.4x more raw compute, so should have 2.7x more in effective compute, or act like a compute optimal model that's 1.6x larger, trained on 1.6x more tokens. So the distinction is smaller than 180 vs. 800.