Thanks for the awesome post! This was a great read, and I’m super excited to see where the work goes next.
I’m curious if you guys have been thinking about belief states/belief state geometry in the context of generalizing models, i.e. those that are trained to be “task learners” rather than “task retrievers”. I am thinking specifically of work on in-context learning in settings like Markov chain mixtures and in-context linear regression, where task-diversity thresholds seem to separate memorizing and generalizing solutions. In the memorizing regime, I migh... (read more)
Thanks for the awesome post! This was a great read, and I’m super excited to see where the work goes next.
I’m curious if you guys have been thinking about belief states/belief state geometry in the context of generalizing models, i.e. those that are trained to be “task learners” rather than “task retrievers”. I am thinking specifically of work on in-context learning in settings like Markov chain mixtures and in-context linear regression, where task-diversity thresholds seem to separate memorizing and generalizing solutions. In the memorizing regime, I migh... (read more)