Interesting post, thanks.
Why should we trust an agent with integrity more than one that is compliant with rules?
This seems too strong to me. At the end you say that 'Integrity doesn’t speak to the goodness of values', but it seems like in the rest of the post you're not really taking that into account. Integrity does seem important to me, and I appreciate the pointer to it (and Velleman) as a useful framing for an important property. But it seems somewhat orthogonal to the question of what values are, and integrity alone says very little about whether we can trust an agent (to be clear, I do think that the Claude constitution specifies values). As a result, passages like the quoted one above seem misleading.
The constitution’s values currently exist in natural language with no formal account of what makes something count as a value, how values relate, or how they should be revised. The aforementioned breakdown of honesty is moving in the right direction. But it still lacks a type system.
The alternative is structured representations that specify the grammar by which values can be expressed, compared, and updated
At the risk of being an over-literal programmer, even after skimming the full-stack paper, I have no idea what this means. Is there somewhere that you give concrete examples of a a type system for values, or an appropriate structured representation, or (from the paper) a grammar for values? It seems like you're drawing on terms from computer science and programming language design (unless that's coincidental) but I don't understand what those terms mean in this context.
Thanks!
For an example of a "type system for values" (agree it's an imperfect computer sciency reference), check out the notion of attentional policies and values cards in this paper: https://arxiv.org/pdf/2404.10636
What we mean is that many types of things commonly gets called "values": matters of taste, social norms, ideological slogans, etc. You may have a preference for "mini skirts" or "brownies over cookies", but most people would agree those aren't really values in the same way "honesty" or "creativity" are; they don't say anything substantial about how you want to live, or what you think is important in life.
but words like "honesty" are too loose; one person may think honesty is never lying, for someone else honesty means "speaking from a place of authenticity at all times", for someone else it's something about epistemic humility.
Hence the need for a tighter type system than just strings, if we want to structurally reason over these.
Published a post about integrity as a frame for trustworthiness in AI alignment, and how it relates to Claude's new constitution.
Cross-posting as I think this might interest folk here. All feedback welcome!
In October 1962, Soviet submarine B-59 was trapped underwater by American destroyers dropping depth charges. The captain, cut off from Moscow and believing war had begun, wanted to launch a nuclear torpedo. Launch required three officers. One of them, Vasily Arkhipov, refused, and nuclear war was averted.
A few months earlier in Jerusalem, Adolf Eichmann was put on trial for organizing the logistics of the Holocaust. His defense, famously documented in Hannah Arendt’s “Eichmann in Jerusalem: A Report on the Banality of Evil”, was that he was simply following orders. He viewed himself as a victim of a system he didn’t design.
These two examples are extremes and not directly analogous with misaligned AI. But they illustrate something that matters in less dramatic contexts too. AI will likely operate within unforeseen situations its rulebook never anticipated (as illustrated by the “moltbook moment”), or in misaligned incentive structures (the market arguably already is one).
In such cases, you need agents that have what Vasily had, and what Eichmann lacked.
Model Integrity
A year ago, we wrote a blogpost arguing that integrity is a more powerful frame than compliance for thinking about trustworthiness in LLMs.
Since then, the labs have diverged in ways that highlight this distinction. OpenAI has committed to compliance—their alignment researcher Boaz Barak published an essay aptly titled “Machines of Faithful Obedience”, and their Model Spec operationalizes a long list of binary rules.
Anthropic has gone a different direction. They recently released a new 84-page constitution. It’s not a constitution in the legal sense, but a document describing in detail what constitutes Claude’s character.
This post makes the case that Anthropic’s move is the better one from the perspective of integrity. To explain why, I’ll draw on the philosopher J. David Velleman, who offers a relevant account for what integrity means. In brief, he argues you can’t have integrity without first having a character to be true to. If that’s right, then Anthropic’s bet on giving Claude a coherent character matters.
Integrity and Character
What does it mean to have integrity?
Velleman argues integrity is about being consistent with who you are. He puts it as follows:
Integrity means acting in ways that cohere with that image. When your actions strain against your self-conception, you could be said to be acting “out of integrity”.
Velleman further develops this with a metaphor from improvisational theater. Think about what makes good improv. An actor playing a character doesn’t do random things. They do things that make sense given that character’s wants, values, habits and personality.
When talking about Claude having a “character,” I’m not making claims about consciousness or moral patienthood. Character, in Velleman’s sense, is a functional property.
It’s also worth mentioning that integrity-as-coherence by Velleman’s account doesn’t guarantee moral goodness. Models also need to be trained on good values in the first place.
There is some reason to believe that models can infer broadly good values provided we point them in the right general direction.3 However, this doesn’t let us off the hook. Organizations fine-tuning models with RL for narrow institutional goals may cause models to drop their character entirely. And even without external pressure, good values that don't cohere or run deep may leave models prone to being manipulated by clever prompting (jailbreaks), or interpreting vague directives in unintended ways (as when Grok took 'maximally based' as license to method-act Hitler).
These can all be seen as failures of integrity, and might be avoided by training the model to have a strong, stable character it can be true to. More work is required to verify this empirically, though.
Integrity and Trust
Why should we trust an agent with integrity more than one that is compliant with rules?
In Self to Self, Velleman uses the classic prisoner’s dilemma to explain why.4 Standard game theory says betrayal is rational. The exception is if you expect to interact again, because then the other party can retaliate and cooperation becomes worthwhile.
A rational agent’s cooperation is therefore conditional. It depends on whether the game repeats, whether anyone can observe what happened, whether there are consequences. Change the circumstances and the behavior changes too.
An agent with integrity cooperates because this is the kind of agent she takes herself to be, not because cooperation happens to pay. Situational circumstances such as whether this is a one-shot game or whether anyone is watching don’t matter, because they don’t change who she is.
The prisoner’s dilemma is a toy example, but the above example points to situations where small changes in context can flip which actions seem justified. For example, imagine an AI agent negotiating contracts on your behalf. Midway through, it discovers the counterparty has made an error in the terms, one that would unfairly benefit you if left uncorrected. A rule-following agent may exploit the mistake, since nothing in its instructions forbids doing so. An agent with integrity does not, because winning by exploiting another party’s error is not the kind of action it can recognize as its own. You could try to encode “don’t exploit mistakes” as a rule, but that only pushes the problem one step upstream, to the next unforeseen situation.
Principles and Character
Anthropic’s previous constitution was in many ways insufficient. In our previous blogpost, we argue that words like helpful, harmless and honest alone won’t lead to model integrity.
The problem with vague values is that they create wiggle room to bend things, for instance, towards the market pressures on AI companies to maximize engagement, retain users or avoid controversy. A loose notion of “helpfulness” can accommodate all of these. Being agreeable is helpful. Telling users what they want to hear is helpful. Avoiding hard truths that might make someone close the tab—that’s helpful too, isn’t it?
We’ve seen this before. Corporate values like “Don’t be evil” often start as genuine aspirations and end as rubber stamps for what was going to happen anyway.
Rules aren’t bad. They have their place as a legible lower-bound safeguard for harmful behavior. But as I’ve already mentioned, for the many situations rules will inevitably fail to anticipate, integrity confers a deeper kind of trustworthiness.
Anthropic’s new constitution doesn’t list rules, and doesn’t rely on vague values. Instead, it gives detailed descriptions of what values like honesty, harmlessness and helpfulness look like in practice. For example, it describes honesty in terms of seven components:
This specification of honesty shares something with our notion of attentional policies, describing what an agent is attending to when enacting a certain value, like honesty. It’s reasonable to expect that an agent trained this way would develop more coherent values, and with them something closer to a stable sense of character and integrity.
Verifying Integrity
How do we know when a model has integrity?
This is an open research question. We can partially test this behaviorally—does the model apply its values consistently across new situations? Does it maintain coherent reasoning under adversarial pressure? Anthropic’s recent paper on disempowerment patterns is a good start. There might also be interpretability approaches that let us evaluate the cohesiveness of a model’s character.
Evaluating whether a model exhibits integrity is one problem. Governing and revising the values it is meant to embody is another. Boaz correctly argues that this is easy when it comes to rules:
How do you contest a character? How do you trace a mistake back to the value that produced it? The constitution’s values currently exist in natural language with no formal account of what makes something count as a value, how values relate, or how they should be revised. The aforementioned breakdown of honesty is moving in the right direction. But it still lacks a type system.
The alternative is structured representations that specify the grammar by which values can be expressed, compared, and updated—without dictating which values to hold. In our full-stack alignment paper, we call this thick models of value. This is, I think, one of the biggest opportunities for improvement in the current constitution.
Conclusion
Integrity isn’t everything in AI alignment. We want models with domain expertise, with good values, with the wisdom to enact them skillfully. Integrity doesn’t speak to the goodness of values. But it does speak to how deeply they run, how stable they are under pressure. It’s what lets us trust a model in situations we never anticipated.
Anthropic’s new constitution is a big step in this direction: it doesn’t just outline slogans or rules, but a character, a self, to be true to. The document reiterates this point poetically in the very last line:
Velleman would agree.
Thanks to Ryan Lowe for careful feedback, and Joe Edelman for helping shape these ideas.