How do you think your observation connects to metacognitive capabilities (not) present in current LLMs? I could imagine they fail at a lot of simple tasks because they are still missing an embedding in this world, especially during training. For sure, your agents are in a harness, as they are during a lot of RLVR training, but this is only very narrow access to this world. They learn to maximize some reward, but the reward is a proxy for some behavior, not for some cognition. There is a striking difference between "I try hard to not fail at that task" and "I am new to this team, I don't want to make errors, but if I do my senior colleague will correct me and next time I'll do better". This is not (only) a question of memory. It is the ability to not perceive oneself outside of a task but rather to see the mutual effects between it and oneself.

I think that LLMs already have some of these metacognitive abilities, like GLM-5.3 reasoning about how its behavior will shape later checkpoints of itself, but they are rather crude. And I think the main reason is that we learn most of our metacognitive abilities not by reward (as it would be for some social or very elementary behavior) but by reflection. Or, put in another perspective, reward shapes behavior while reflection shapes (meta-)cognition. For sure these two dimensions aren't independent, but they are not always tied together. As far as I know there exists no real training by reflection for LLMs. Constitutional training still trains the right policy directly; the 'reflection' is just the step to create this policy.

One of the rare examples of something coming close to real reflection, although still supervised, seems to be counterfactual reflection training. Instead of shaping a policy by RL, the model is fine-tuned on (self-generated but supervised) reflections on counterfactual situations. These reflections generalize beyond these counterfactual situations (and also to the behavior). I suspect this is the LLM learning a better self-model. But it is still far away from being able to see oneself embedded in a context and weighing whether the juice box or the bleeding finger needs more attention (I would expect a model with better metacognitive abilities to reason more like: "Ok, my juice box won't open. I should get it open, but this is probably a task of low importance, so the problem isn't urgent and I shouldn't present it as an emergency". RL on opening juice boxes wouldn't allow the model to see this task as less important than anything else it has been RL on).