There's something called the No Free Lunch theorem which says, approximately, that there's no truly general algorithm for learning: if an algorithm predicts some environment better than chance, there must exist some adversarial environment on which it will do at least that much worse than chance. (Yes, this is even...
Epistemic status: slightly pruned brainstorming. I've been reading John Wentworth's posts on the natural abstraction hypothesis recently, and wanted to add some of my own thoughts. These are very much "directions to look in" rather than completed research, but I wanted to share some of the insights central to my...
Epistemic status: speculative. First, a few paper titles: * Pretrained Transformers as Universal Computation Engines * Can Wikipedia Help Offline Reinforcement Learning? (Answer: yes.) * Pretrained Transformers Improve Out-of-Distribution Robustness * Data Distributional Properties Drive Emergent In-Context Learning in Transformers The gist of the first three studies is that transformers...