This is a special post for quick takes by anaguma. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
From what I understand that's basically correct; you go to depth recurrence when you're bottlenecked by memory or training data but still want to increase compute.
With the memory situation being what it is, I can see how a bit of depth recurrence might become attractive.