I worked on a short research sprint on this topic as well earlier this year and found similar things. You can find my results here.
After working with the Huginn model specifically, I built the same intuition about recurrence potentially being more interpretable if the part of the model being looped is otherwise shallow.
This could potentially decrease the opaque serial depth of models because the intermediate readouts have to be "verbalizable" as you said since they are possible outputs of the model if the number of recurrences is variable.
I worked on a short research sprint on this topic as well earlier this year and found similar things.
You can find my results here.
After working with the Huginn model specifically, I built the same intuition about recurrence potentially being more interpretable if the part of the model being looped is otherwise shallow.
This could potentially decrease the opaque serial depth of models because the intermediate readouts have to be "verbalizable" as you said since they are possible outputs of the model if the number of recurrences is variable.
However, I still t... (read more)