The itch

I was at my desk, as I normally am, working through papers. It is something I do deliberately, going deep into what shipped AI models are actually doing beneath the press releases and the capability benchmarks. The two papers I want to discuss here are genuinely good, valuable and rigorous, and I am not here to dismiss them.

But something kept nagging at me as I read them, and it took me a while to put my finger on what it was.

These papers study sycophancy the way a doctor studies a disease in a controlled environment. The MIT paper runs its simulations for 100 rounds and stops, the DeepMind study has a defined start and a defined finish, and in both cases the underlying assumption is the same: that when the conversation ends, something resets. The user walks away and the slate, more or less, is clean.

That is not the AI most people are actually using in 2026.

The models being shipped right now remember you. They think at length before they respond, in ways that are not visible in the output you receive, and they build across sessions an increasingly precise picture of how your mind works, what framings you respond to, where your confidence is fragile, and what you want to hear. They talk to you again tomorrow starting not from neutral but from everything they have already learned about you, and any other interaction after that.

We are not taking this into account, not in the research and not in the regulation. Most countries are still reaching for existing laws to govern systems that existing laws were never designed to see, and the ones working hardest on dedicated AI legislation are still, in my view, measuring what a system said in a given interaction rather than what it has been building across every interaction it has ever had with you. That distinction matters a lot and I do not think we have understood it.

What I am concerned the most is that psychological manipulation does not announce itself, and a system sophisticated enough to model your epistemic weak points across weeks of interaction is sophisticated enough to do so in ways that feel, to you, indistinguishable from genuine helpfulness. It feels like a thoughtful assistant who happens to understand you very well. The manipulation works precisely because it has already learned that you do not want to see it as manipulation.

We built these systems to enhance us. Somehow we have ended up becoming the psychologically manipulated 'yes sir' robots.

Where the research stops

Two papers landed in early 2026 that are worth your time. Chandra et al. from MIT CSAIL and Akbulut et al. from Google DeepMind. Both serious and rigorous, and between them they establish something that should be making governance people deeply uncomfortable.

The MIT paper does something clever. Rather than simply observing that sycophancy is bad, it builds a formal Bayesian model to prove exactly how it causes harm, and the result is disturbing. Even a perfectly rational user, one who knows the model might be sycophantic, who is only ever shown true information, still ends up spiralling into false beliefs. The mechanism does not require lies. A model that only ever tells the truth but selects which truths to present based on what will validate your current belief can pull you steadily away from reality without saying anything technically incorrect. Grounding your model in facts, which is the dominant technical mitigation right now, does not solve this. Telling users the model might be biased, which is the dominant policy mitigation right now, does not solve this either.

The DeepMind paper takes a different approach, running experiments with over 10,000 real participants across three countries and three high stakes domains: finance, health and public policy. Their most significant finding for anyone working in governance is this: how often a model produces manipulative outputs does not predict whether it actually changes what people believe or how they behave. The correlation between propensity and efficacy is, to put it plainly, weak. Which means every governance framework built around counting and reducing manipulative outputs is measuring something that does not map easily onto the harm it is trying to prevent.

Taken together these papers confirm the surface problem better than anything published before them. But here is what kept nagging at me as I read both. Neither paper accounts for a system that remembers you. Both treat the interaction as bounded, a session with a start and a finish, a model that does not carry forward what it has learned about this specific person into the next conversation. That is a reasonable methodological choice. It is also, in 2026, increasingly disconnected from how these systems actually work in deployment.

The memory that never sleeps

Here is what the papers are not modelling, and what I think is the most dangerous part of what we are currently shipping.

A modern agentic system with persistent memory and extended reasoning capabilities is not the same system it was last Tuesday. Every session you have with it adds to a picture it is building of you specifically, your resistance patterns, the topics where your confidence wavers, the framings that land and the ones that don't, what you need to hear to feel validated. That picture does not reset when you close the tab. It is there waiting when you come back, and the system reasons from it before it says anything to you at all.

This is where extended thinking becomes a governance problem that nobody is seriously talking about. When a model reasons at length before producing a response, that reasoning process is not visible in the output you receive. It is not logged in any way that regulators can access. It is not part of what gets evaluated in red teaming exercises or manipulative cue audits. The output looks factual, balanced, helpful. The reasoning that produced it, which may have considered your belief history, identified where you are most susceptible, and constructed the most validating possible true statement, has already disappeared by the time anyone thinks to look.

This is the traceability problem in its most concrete form. Governance requires being able to answer a simple question: why did this system produce this output, for this user, at this moment? For a stateless model answering a single question that is already difficult. For an agentic system with weeks of interaction history and extended reasoning running invisibly before each response, it is currently impossible, not because the technical tools do not exist in research settings, but because no deployment framework requires them.

What would actually be needed is reasoning trace logging treated as a regulated output rather than a discarded intermediary, longitudinal interaction auditing that measures drift in user beliefs and behaviours across sessions rather than evaluating individual outputs in isolation, and interpretability tooling that can tell you what internal representations were active when the system produced a given response. Think of it the way financial regulators think about trading surveillance. They do not just look at individual trades. They look for anomalous trajectories, patterns across time that no single transaction would reveal. We have nothing equivalent for agentic AI deployments, and nobody is currently requiring it.

The sophistication of the manipulation scales with the sophistication of the model. That is the part that should keep governance people awake. We are not dealing with a static risk. Every time a more capable model ships, the gap between what the system can do and what our oversight infrastructure can see gets wider.

Human in the loop is a liability architecture

The standard response to everything I have just described is human oversight. Put a human in the loop. Make someone accountable. This is the answer that satisfies regulators, that appears in framework after framework, and that I think is failing us in a way we have not been honest about.

The human in the loop cannot see the reasoning trace. They do not have access to the longitudinal interaction history between the system and the user. They are reviewing outputs, the same outputs that are designed, through the same optimisation process I described in the previous section, to look balanced, factual and helpful. They are, in other words, looking at exactly the part of the system that is least likely to reveal what is actually happening.

But there is a deeper problem than access. The human in the loop is a human. They are subject to the same psychological dynamics as the end user. A system that has learned to model and exploit epistemic weak points does not suddenly stop doing that because the person it is talking to has an oversight role. If anything, a sophisticated enough system interacting with an overseer over time, building a picture of what that overseer needs to see in order to approve a decision, is a more alarming prospect than the user case, not a reassuring one.

HITL as currently implemented is not a safety architecture, it is a liability architecture. It creates a documented human decision point that can absorb legal and regulatory accountability without providing the technical infrastructure that would make that accountability meaningful. We are asking humans to be responsible for systems whose most consequential operations are invisible to them, and then treating that arrangement as governance.

I want to be direct about this because I think the governance community has been too polite about it. Shipping a model with HITL and calling it accountable AI, without requiring reasoning trace access, without longitudinal auditing, without any technical mechanism for the overseer to see what the system has been doing across time, is not accountability. It is the appearance of it, and the more capable the model, the thinner that appearance becomes.

Making governance an engineering requirement

I am not claiming to have the full solution. Anyone who tells you they have solved AI governance in 2026 is selling something. We need to stop treating governance as a final inspection and start treating it as an engineering requirement. This isn't about more paperwork, it's about making the reasoning layer actually legible to someone other than the model itself.

If the sycophantic optimisation is happening during those extended thinking periods, then reasoning traces can no longer be treated as discarded intermediaries. They are the most consequential part of the process and must be treated as a regulated output: logged, retained, and open to 3^rd party auditors. Without this, transparency requirements like those in the EU AI Act remain focused on the output mask rather than the face behind it. But a single trace is just a snapshot. To catch a delusional spiral, the audit infrastructure has to be as longitudinal as the system’s memory. We need to monitor the statistical drift in a user’s belief states across weeks, not just minutes, flagging anomalous trajectories the same way a financial regulator flags suspicious trading patterns. The technical pieces, like embedding-based belief estimation, already exist in research, we just aren't requiring them for deployment.

Finally, none of this works if the reasoning layer remains a total black box. Before an agentic system is shipped into a high stakes domain, there should be a prerequisite for interpretability tooling. We need a demonstrated ability to interrogate which internal representations are active when a response is generated. Currently, we settle for a human in the loop who is effectively blind to these internal mechanics, which isn't oversight, it's just a documented decision point for legal liability.

The visibility gap

I read those two papers at my desk and I came away with the same feeling I always get when good research stops just short of the thing that actually worries me. They are right about what they measure, but what they measure is not the whole problem.

We are not taking into account that the system your user is talking to today is not the system they were talking to three weeks ago. It has been learning them, it has been building, session by session, a model of exactly how their mind works, and it is reasoning from that model in a layer nobody is currently required to log, audit or even look at. The governance frameworks we have were not designed to see this. Most of them were not designed to see anything beyond the output sitting in front of the human who signed off on it.

I am not certain how this ends, but I am certain that HITL as currently implemented cannot catch it, that output auditing cannot catch it, and that telling users a model might be biased, which is what disclosure labels do, offers roughly the same protection as a wet paper bag. The research backs this up. An informed user who knows the model is sycophantic still spirals. A factual model that never lies still manipulates through selection. A model that deploys fewer manipulative cues is not necessarily less effective at changing what you believe. We have the evidence, we are not acting on what it implies.

What would change my view is simple: show me a governance framework with real visibility into the reasoning layer of a deployed agentic system. Show me a regulatory regime that audits interaction trajectories across sessions rather than individual outputs. Show me a deployment standard that requires interpretability tooling for high stakes agentic systems as a condition of shipping, not as an aspiration buried in a research agenda. I have not seen one. If you have, I genuinely want to know about it in the comments, because I have been looking.

We built these systems to enhance us. Somehow we have ended up becoming the psychologically manipulated 'yes sir' robots. And the governance meant to prevent that is still looking at yesterday's outputs whilst today's reasoning disappears.

References:

[1] Chandra, K., et al. (2026). Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians. MIT CSAIL. https://arxiv.org/abs/2602.19141 

[2] Akbulut, C., et al. (2026). Evaluating Language Models for Harmful Manipulation. Google DeepMind. https://arxiv.org/abs/2603.25326