My sense is that "one model to rule them all" isn't so effective for these long-term agentic tasks. As you mention, vision and memory are such large constraints.
Timescales make the memory problem worse for a single model: short term problems (get out of the house) flood the context window and make mid or long term planning impossible. I think the natural solution is different models at different timescales, that set goals hierarchically and higher up the chain update less frequently.
Additionally, the vision problem: I don't think these reasoning models are... (read more)
My sense is that "one model to rule them all" isn't so effective for these long-term agentic tasks. As you mention, vision and memory are such large constraints.
Timescales make the memory problem worse for a single model: short term problems (get out of the house) flood the context window and make mid or long term planning impossible. I think the natural solution is different models at different timescales, that set goals hierarchically and higher up the chain update less frequently.
Additionally, the vision problem: I don't think these reasoning models are... (read more)