This post's focus on accountability kind of implies a continuous model of integrity, and I think there's an important near-discrete component which is, whether you're trying to use beliefs to navigate the world, or just social reality. Nightmare of the Perfectly Principled explored this a bit; enforcing consistency in a way that's not about trying to use beliefs doesn't help that much. For people who are trying to use their mental structures to model the world, pressure towards inconsistency doesn't register as a temptation to be nobly avoided, it registers as abuse and gaslighting.

This post seems excellent overall, and makes several arguments that I think represent the best of LessWrong self-reflection about rationality. It also spurred an interesting ongoing conversation about what integrity means, and how it interacts with updating.

The first part of the post is dedicated to discussions of misaligned incentives, and makes the claim that poorly aligned incentives are primarily to blame for irrational or incorrect decisions. I’m a little bit confused about this, specifically that nobody has pointed out the obvious corollary: the people in a vacuum, and especially people with well-aligned incentive structures, are broadly capable of making correct decisions. This seems to me like a highly controversial statement that makes the first part of the post suspicious, because it treads on the edge of proving (hypothesizing?) too much: it seems like a very ambitious statement worthy of further interrogation that people’s success at rationality is primarily about incentive structures, because that assumes a model in which humans are capable and preform high levels of rationality regularly. However, I can’t think of an obvious counterexample (a situation in which humans are predictably irrational despite having well-aligned incentives for rationality), and the formulation of this post has a ring of truth for me, which suggests to me that there’s at least something here. Conditional on this being correct, and there not being obvious counterexamples, this seems like a huge reframing that makes a nontrivial amount of the rationality community’s recent work inefficient-if humans are truly capable of behaving predictably rationally through good incentive structures, then CFAR, etc. should be working on imposing external incentive structures that reward accurate modeling, not rationality as a skill. The post obliquely mentions this through discussion of philosopher-kings, but I think this is a case in which an apparently weaker version of a thesis actually implies the stronger form: philosopher-kings being not useful for rationality implies that humans can behave predictably rationally, which implies that rationality-as-skill is irrelevant. This seems highly under-discussed to me, and this post is likely worthy of further promotion solely for its importance to this issue.

However, the second broad part of the post, examining (roughly) epistemic incentive structures, is also excellent. I strongly suspect that a unified definition of integrity with respect to behavior in line with ideology would be a significant advance in understanding how to effectively evaluate ideology that’s only “viewable” through behavior, and I think that this post makes an useful first step in laying out the difficulties of punishing behavior unmoored from principles while avoiding enforcing old unupdated beliefs. The comment section also has several threads that I think are worthy of revisitation: while the suggestion of allowing totally free second-level updating was found untenable due to the obvious hole of updating ideology to justify in-the-moment behavior, the discussion of ritual around excessive vows and Zvi’s (I believe) un-followed-up suggestion of distinguishing beliefs from principle both seem to have real promise to them: my guess would be that some element of ritual is necessary to avoid cheapening principle and allowing for sufficient contradictory principles to justify any behavior.

Finally, the discussion of accountability seems the least developed, but also a useful hook for further discussion. I especially like the suggestion of “mandatory double-crux”‘powers: I’ve informally tried this system by double-cruxing controversial decisions before action and upon reflection, I believe it’s the right level and type of impediment: likely to induce reflection, a non-trivial inconvenience, but not a setting that’s likely to shake well-justified beliefs and cause overcorrection.

Overall, I support collation of this post, and would strongly support collation if it was updated to pull more on the many potential threads it leaves.

I tend to think of integrity as the ability to have true beliefs and take good action in spite of incentives. I'm thinking of the person who chooses not to break a principle, even when nobody is looking and it's not an obviously important case of the principle and there's a lot of moral/personal value to be gained from it.

But there is also the second part. A person with high integrity can make many principled decisions in spite of incentives, but a person with high integrity also notices when they're entering an environment where they're going to be faced with too many decisions to be able to make good ones.

For example, I have a personal rule against googling a large class of gossipy/political things that feel yummy except when I have explicitly time-boxed time to think through it, because at the minute I don't trust my background reasoning processes to incorporate the evidence in an unbiased way. I used to google gossipy/political topics a fair bit. Sometimes I still do it at midnight when I am bored and not tired. But I increasingly have been able to catch myself and say "this is not behaviour I endorse", even though it's quite difficult because there's not an alternative yummy thing to do. Increasingly, I'm becoming someone who can say "well, I made a policy decision to not take this action, so I won't". However the more useful thing is noticing that my phone is generally giving me a lot of decisions to make that are difficult and often many of the choices I don't endorse, and then systematically remove those options. I've done things like blocked social media on my phone except for 2 hours on Saturday, and blocked it permanently on my laptop, and followed some of Tristan Harris's advice on organising my apps. Both of these things preserve my ability to think clearly and take good action.

There's this idea that people with integrity can be handed power and be expected to continue doing the sort of things they did when they had less power - be the same sort of person, hold the same principles, etc. Or alternatively they will turn down the power if they think that they won't be able to be the same person. Following that, there's the old idea that the people who should be given power are those who don't want it. I'm not sure this really holds up - those who don't want it often have actual models predicting that they will experience failures of integrity. Though at least they have a model of where the mistakes will come and can try to prepare for them. Most people don't even know where their errors will come.

I'm trying to figure out whether "acting in accordance with your stated beliefs" feels like the right description. I guess that there's this thing relating to noticing when you will stop being the same kind of person, and avoiding taking that action unless you endorse it. I expect a person with a lot of integrity to change things about themselves and their actions, but in ways that they reflectively endorse, rather than being pulled around by the winds of the local incentives.

If I am to propose an alternative definition, it's that someone with integrity is someone I can trust to follow their long-term goals and not be thrown off-course by following short-term incentives. Someone who is able to turn down power when it will throw them off their long-term course, even if they haven't figured out how to get power a different way yet. Someone who will learn to say no to the short-term offers they are getting.

This means that highly competent people in positions of power often have less accurate beliefs than much less competent people who are not in positions of power.

Not sure how strong you intend this statement to be (due to ambiguity of "often"), but I would think that all-else-equal, a randomly selected competent person with some measure of power has more accurate beliefs than a less competent person w/o power, even after controlling for e.g. IQ.

Would you disagree with that?

I'd grant that the people with the very most accurate beliefs are probably not the same as the people who are the very most competent, but that's mostly just because the tails come apart.

I'd also grant that having power subjects one to new biases. But being competent and successful is a strong filter for your beliefs matching reality (at least in some domains, and to the extent that your behavior is determined by your beliefs), while incompetence often seems to go hand-in-hand with various kinds of self-deception (making excuses, blaming others, having unrealistic expectations of what will work or not).

So overall I'd expect the competent person's beliefs to be more accurate.

I like this part best:

Choose carefully who, and how many people, you are accountable to

Importantly, these mostly won't be individuals. Instead, we mostly have groups, and the composition of those groups is not subject to our decisions, eg: our own families, our spouse's families, the other employees at work, the other congregants at church, the other members of the club, etc.

I feel strongly that selecting and acting within groups is a badly neglected area of moral reflection.

For a long time, I viewed integrity as a virtue. For the past 15 years or so , I've created and strived to live by mission statements that elucidate my values. I had a strong sense of my ideal self, and a strong sense of my "actual self" and worked to live up to it as often as I could.

More recently, the concept of my "self" has started to seem less of a coherent concept, and the concept of an "ideal self" has started to seem less obvious to me. Instead, I've started to shift towards context-specific integrity, that is, being legible and consistent to specific groups of people, but not having a global set of principles that I live up to. This allows me to focus more on terminal values and things I care about, while still being predictable to outside people.

There's a sense where there's a global type of integrity which is something like "Am I actually acting in accordance with my current frame of meaning and the things I care about?" but there's no way to put that into a single consistent set of principles.

I mostly agree with the direction you're going, but I find myself with something to disagree about in almost every point. I can't tell if these are irrelevant nitpicks, or if I'm getting a hint that some assumptions underlying the text are very different from mine. Two examples:

1) The title "Integrity and accountability are core parts of rationality" confuses me. Integrity and accountability are core parts of social and personal interaction, and important topics in the culture of any group (including rationalists). I don't know that they're core parts of rationality itself. Our disagreement here may be that I don't think "group rationality" is a thing. Rationality is individual, and some (perhaps all) groups would benefit by having more rational members and more rational behavior by members, but ones values and beliefs are fundamentally private and only partially communicable.

2) "This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized. " I like and agree with the observation, and I'm suspicious of the causality. I think we'll need to dive into categorization and legibility (to the incented and to the observer) of incentives to untangle what this means and when it's true.

A decent solution to the "who should you be accountable to", from the wisdom of the ancients (shows thought on many of the considerations mentioned)

When in doubt, remember Warren Buffett’s rule of thumb: “… I want employees to ask themselves whether they are willing to have any contemplated act appear the next day on the front page of their local paper—to be read by their spouses, children and friends—with the reporting done by an informed and critical reporter.”

…when I say “integrity” I mean something like “acting in accordance with your stated beliefs.” Where honesty is the commitment to not speak direct falsehoods, integrity is the commitment to speak truths that actually ring true to yourself, not ones that are just abstractly defensible to other people. It is also a commitment to act on the truths that you do believe, and to communicate to others what your true beliefs are.

your handle of “integrity” seems to point at something quite similar to the thing at which joe carlsmith’s handle of “sincerity” (https://joecarlsmith.com/2022/12/23/on-sincerity) points. am i missing something, or are these two handles for ~the same thing?

I have come to believe that people's ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain.

I think this can be read many ways. First, obviously if a person is subject to an incentive to hold true beliefs about X, they will start trying to learn about X and their beliefs will become more accurate. This part isn't very interesting.

The more interesting parts of your idea, I think, are the notions that

(1) In the absence of incentives to have true beliefs about X, people don't just have no beliefs about X, but in fact tend to have beliefs that are wrong.

(2) In the presence of incentives to have wrong beliefs about X, people tend to adopt those wrong beliefs.

I'm less convinced that these things are true generally. I do think they are true of many people if we define "belief" as "an opinion that a person expresses". But whether that's a reasonable definition of belief is unclear---I think that often the people for whom (1) and (2) are true are the same people who don't care whether their expressed opinions are correct. In that case the observation reduces to "if people don't care about saying true things, they will say things they are incentivized to say", which isn't surprising.

For the average LessWrong reader, I'm not convinced (1) and (2) are accurate. The observation that people tend to have beliefs that align with their incentives might instead be explained by a tendency for people with belief X to gravitate towards a position that rewards them for having it.

I have come to believe that people's ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain.

Are good ways of incentivizing accurate models the same across domains? (As opposed to different, say based on differences between the domains, or their practitioners.)

I think of integrity as a more advanced form of honesty – when I say “integrity” I mean something like “acting in accordance with your stated beliefs.” Where honesty is the commitment to not speak direct falsehoods, integrity is the commitment to speak truths that actually ring true to yourself, not ones that are just abstractly defensible to other people. It is also a commitment to act on the truths that you do believe, and to communicate to others what your true beliefs are.

So integrity is three things, and it's having all three of them, at the same time.

In the context of incentive design, I find thinking about integrity valuable because it feels to me like the natural complement to accountability. The purpose of accountability is to ensure that you do what you say you are going to do, and integrity is the corresponding virtue of holding up well under high levels of accountability.

Four things. This part of "integrity" sounds like being able to hold up under stress, and being good with people.

Integrity can be a double-edged sword. While it is good to judge people by the standards they expressed, it is also a surefire way to make people overly hesitant to update.

This is surprising, because it seems like usually people are judged for the standards they express, not by the standards they express. Both seems like a 'surefire way to make people overly hesitant' to a) share their standards, b) actually do this integrity thing.

I think a generally better setup is to choose a much smaller group of people that you trust to evaluate your actions very closely,

The common sense which makes this virtue seem virtuous, and useful, instead of a disaster.