This is a special post for quick takes by anaguma. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
(I haven't read the article, don't have a subscription, but my understanding is) they should be getting unlimited serial depth by feeding in previous latent states at the bottom of the stack at the next token, while e.g. running autoregression. That thoroughly breaks the previous information-flow story.