LESSWRONG
LW

Comment Permalink
Said Achmiz4mo192

Problems with corrective methods:

To a first approximation, people do not change.

To a second approximation, people only change by their own volition, but cannot be changed otherwise.

Why should anyone who is not already “the right sort of people” to fit into your system, wish to change so as to make themselves that sort?

It can only be due to deep value alignment with your systems’s goals.

But now you have a selective task: to find those whose values are deeply aligned with those of your system. You import all the problems of selective methods, then; and add to that, the challenge of correction.

Without value alignment, you can incentivize volition toward change. But what you actually incentivize is the appearance of that change, or some approximation. Is it good enough? It might be, in some cases. Some tasks are more demanding than others; approximation may serve.

Correction requires an existing status differential. Without such, it is perceived as a bid for greater status than you (or your system) have already. People resist such bids.

Even assuming away all of the above, correction is hard. It takes a greater degree of mastery to teach a skill than to use it or to recognize it.

People are different; they vary in potential. To assume that anyone, even granting the most sincere and fervent wish to change, may become as they must be in order that they may play a role in your system, is to set yourself up for disappointment and failure. But recognizing this, you must again select. (On the other hand, it may be an easier selection task… or it may not be.)

LVSN4mo38

Your group's collective belief or disbelief in correction is self-fulfilling prophecy.

See in context
by Said Achmiz
2 min read5th Mar 202311 comments

94

Coordination / CooperationMechanism DesignPracticalWorld Optimization
Frontpage

(A reference post for a concept that comes up often enough to warrant such a thing.)

If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary.

—James Madison

Three kinds of methods

The one comes to us (wishing to build a social system, and having only people, and no angels, to work with) and says:

“My system would work perfectly, if only everyone involved would behave in the optimal manner!”

Granted; unfortunately (or fortunately), not all people can be relied on to behave optimally. How to make the system work despite this?

There are three sorts of approaches:

  1. Selective methods—build your system out of only the right sort of people, and exclude the wrong sort.
  2. Corrective methods—apply such measures as will make the people in your system alter their behavior, to conform to relevant optimality criteria.
  3. Structural methods—build your system in such a way that it will work if people behave in the ways that they can be expected to behave.

Examples

Work

The challenge: build an organization (or a team within one) that will be able to accomplish various desirable projects.

  • Selective: hire people who have the skills/experience/etc. to do the work; don’t hire (or fire, if discovered post-hiring) people who aren’t capable of doing the work.

  • Corrective: on-the-job training; social approval/disapproval from coworkers for good/bad work.

  • Structural: bonuses and other financial incentives for performance; technological and process improvements that reduce skill requirements.

World of Warcraft

The challenge: assemble a raiding guild that will be able to defeat the most challenging boss monsters.

  • Selective: accept players who can demonstrate competence in their chosen raid role; exclude those who can’t or won’t perform.

  • Corrective: teach inexperienced players to play better; shame lazy or selfish players into putting in effort, and contributing to the guild’s success.

  • Structural: assign raid members to roles that best fit their talents and inclinations; design a loot distribution system that incentivizes effort and effective participation.

Governance

The challenge: place over society a government, that will rule for the good of all.

Which way is best?

I have no revelatory answer. Probably it varies from one case to another. And—as the examples show—the approaches aren’t mutually exclusive. All three can be combined, potentially, or any two. Each has its advantages; each, also, its drawbacks. (I will explore some of these in the comments section for this post.)

The critical thing, I think, is just to be aware that all three types exist.

Postscript

This post explains, finally—it only took five years!—what I meant by this comment.

Coordination / CooperationMechanism DesignPracticalWorld Optimization
Frontpage

94

New Comment
11 comments, sorted by
top scoring
Click to highlight new comments since: Today at 7:26 AM
[-]Said Achmiz4mo234

Problems with structural methods:

Designing structures and systems is hard. Designing structures and systems that will act upon people, and will channel their behavior into a form consonant with your goals, is very hard.

People are different. Different people will interact with your system in ways different enough that it would be impossible for your design to handle them all.

People have goals of their own. They will exert optimization pressure upon your system, to make their desired results come about; some of these results will be at odds with what you want; and you cannot foresee everything that anyone might do to, with, or in your system, to ensure that people’s behavior has only those results you want.

Make your structures rigid enough to minimize the odds of undesired outcomes, and that same rigidity will constrain you as well. You cannot build your system out of manacles and chains, if you wish anything of any consequence to come of it.

Make your structures flexible, and you make them corruptible.

There is no structure so perfect that none may be found who are so stupid or so evil as to wreck it.

Here you may do well to remember that the real rules have no exceptions. But what are “the real rules” but your (and your system’s) values?

How will you ensure that those remain “the real rules”, except by never giving power within your system to anyone who isn’t aligned with your system’s values?

Now you once again face a selection task.

More problems: designing structures being hard, will you do it on your own? Or will you work with others? How will you be sure that they are suitable? If only the ideal people could be found for the task… and now you’re back to the beginning.

The regress is not infinite, surely, but any systems we have, from which to start the process—the recursive base case—already exist. They are not perfect structures, surely! So, at some point, you must rely on other methods than only structural ones.

[-]Said Achmiz4mo233

Problems with selective methods:

What if it’s hard to tell, in advance, who is “the right sort of people”, and who is “the wrong sort”?

Well, perhaps that’s not so bad, if the decision to include is not irrevocable; if, having included someone, you can easily exclude them later.

But what if it’s not easy to exclude them later?

If you depend on having only the right sort of person within your system, then might a wrong sort of person, included even temporarily, do serious damage?

What if removing someone is very costly?

All of these are only the risks associated with false positives. What about false negatives?

What if it’s hard to identify the right sort of person, and such people are so rare (and/or you need so many of them) that only including those you’ve identified as the right sort means that you miss out on many people whose inclusion would substantially increase your system’s value, and without whom you risk failure?

Also, are there costs imposed by your selection procedure itself? (These costs might be borne by the system and those within it, by prospective members, or by uninvolved others—the latter sort usually being known as “externalities”.)

Such costs may actually be (part of) the selection mechanism—but they aren’t always.

Are there consequences of the use of selective methods, that shape your system in ways other than the simple fact of the methods’ results?

[-]johnswentworth4mo110

Missing what I'd consider the biggest problem: it seems like the vast majority of problems in real-world social systems do not stem from malign or unusually incompetent actors; they stem from failures of coordination, failures of information-passing, failures of anyone with the freedom to act noticing that nobody is performing some crucial role, and other primarily-structural issues. Insofar as that's true, selection basically cannot solve the majority of problems in social systems.

Conversely, well-designed structures can solve selection failures, at least to a much larger extent than selection can solve structural failures. Designing systems to work fine with mostly-average people and be robust to a few negative outliers is difficult, but possible. So there's a real asymmetry between structural vs selective approaches.

[-]Said Achmiz4mo192

Problems with corrective methods:

To a first approximation, people do not change.

To a second approximation, people only change by their own volition, but cannot be changed otherwise.

Why should anyone who is not already “the right sort of people” to fit into your system, wish to change so as to make themselves that sort?

It can only be due to deep value alignment with your systems’s goals.

But now you have a selective task: to find those whose values are deeply aligned with those of your system. You import all the problems of selective methods, then; and add to that, the challenge of correction.

Without value alignment, you can incentivize volition toward change. But what you actually incentivize is the appearance of that change, or some approximation. Is it good enough? It might be, in some cases. Some tasks are more demanding than others; approximation may serve.

Correction requires an existing status differential. Without such, it is perceived as a bid for greater status than you (or your system) have already. People resist such bids.

Even assuming away all of the above, correction is hard. It takes a greater degree of mastery to teach a skill than to use it or to recognize it.

People are different; they vary in potential. To assume that anyone, even granting the most sincere and fervent wish to change, may become as they must be in order that they may play a role in your system, is to set yourself up for disappointment and failure. But recognizing this, you must again select. (On the other hand, it may be an easier selection task… or it may not be.)

[-]LVSN4mo38

Your group's collective belief or disbelief in correction is self-fulfilling prophecy.

[-]MondSemmel4mo40

Let's apply this framework to the Searching for Outliers essay. I suppose the lesson there is that, for sufficiently heavy-tailed outcomes (where, say, some people are a 1000x better fit than the average), selective methods (i.e. searching for outliers) dominate over corrective methods (e.g. better training). And then it talks about how to do selection well.

[-]Said Achmiz4mo40

Yes. This is, indeed, a special case of a general point. The post says:

Corrective methods—apply such measures as will make the people in your system alter their behavior, to conform to relevant optimality criteria.

But of course this comes with a critical assumption: that “optimal behavior” is something that the people in your system are capable of doing. And while that may be a reasonable assumption in many cases, there’s absolutely no guarantee that it will hold in all cases.

For example, no amount of on-the-job training, social approval or disapproval, or even literal torture, will avail you if you hire me to play in your symphony orchestra! (Because I’m tone deaf.) Neither will corrective methods help if you sign Stephen Hawking for the New York Knicks.

These are relatively trivial example, of course—but the principle generalizes better than most people care to think.

For one thing, many sorts of complex behaviors have cognitive requirements; it does not suffice to simply “decide” to do good work, or to play effectively, or to rule effectively—so motivation-oriented corrective methods are inherently limited in their effectiveness. Skill-oriented corrective methods, meanwhile, suffer from the problem that many of them just don’t work in the general case, or themselves have ability requirements.

Furthermore, there is a weaker but even more widely applicable form of this point: that corrective methods may vary in effectiveness depending on the innate abilities of the people involved.

All of this is really an elaboration of this part of my comment on the problems of corrective methods:

People are different; they vary in potential. To assume that anyone, even granting the most sincere and fervent wish to change, may become as they must be in order that they may play a role in your system, is to set yourself up for disappointment and failure. But recognizing this, you must again select. (On the other hand, it may be an easier selection task… or it may not be.)

And, of course—as you suggest—in cases where there is a sufficiently large gradient of ability, corrective methods may be essentially useless without first applying very strong selection.

[-]MondSemmel4mo40

And I suppose a counter-example to the outlier essay would be a fast food chain hiring tons of people for some seasonal increase in demand: here the idea is that workers are presumed to be replacable.

Now selection is much less useful due to the light-tailed skill distribution and high rates of churn. Instead, the organisation is designed such that work is split into simple and clearly defined roles (structural method) which anyone can be easily trained in (corrective method).

[-]Alexander Gietelink Oldenziel4mo43

I like this classification. Clear, crisp, easy to use. I'll see if I will utilize it in my daily thinking.

[-]Said Achmiz4mo40

Of course, the sharp division into three types of methods should not be taken too seriously; nor, and especially, does the division imply that a choice must be made. The methods often overlap and interoperate, to a degree that makes them hard to cleanly separate in practice.

Consider the simplest of the three examples in the post: a World of Warcraft raiding guild. (See my previous posts about rationality lessons from WoW for some background.)

Almost nobody will care about putting in the effort to improve, to learn to play their class/role better, unless there’s some incentive to do so. Likewise, the approbation/disapprobation of their guild mates will mean nothing unless players have some reason to care about their social status within the guild. These incentives and reasons generally are “people who perform badly aren’t invited to raids, or don’t have permanent spots on raid teams; thus they have fewer chances at acquiring loot”.

In other words, what motivation do people have to submit to correction?

Thus corrective methods rely on selective and structural methods.

Meanwhile, saying “assign raid members to roles that best fit their talents and inclinations” sounds properly humane and considerate, but the reality is that some players aren’t good enough to play any useful role in a raid (in some cases, this can be corrected; in others, not).

And, likewise, some people are bad actors. In any system of incentives and rewards, they will devote most of their energy to exploiting the system (and patching to prevent exploits often makes the system worse overall for everyone else).

Thus structural methods rely on selective and corrective methods.

At the same time, it often does no good to try to select players who are “good” in some sense that refers to the final output (raid performance per se).

For one thing, there is such a thing as “culture fit” (yes, even in WoW); someone may fit well into the system—the social environment, the raid organization scheme, etc.—that you’ve created, or fit badly, resulting in a dimension of contribution effectiveness which is not perfectly correlated with “objective” measures of raid performance that can be applied across guilds.

Similarly, because a raiding guild must, by its nature, be able to adapt to new challenges (which means both “new raid content” and “new organizational and task challenges, created by shifting membership and raid composition”), adaptability and the capacity and willingness to learn and improve is very often a more important trait for a prospective raid member than performance on “objective” metrics.

Thus selective methods rely on structural and corrective methods.

And all of these patterns surely manifest in more complex, “real-world”, scenarios.

[-]mukashi4mo32

Added to my Anki. This is very clear framework to think about some problems. Thanks!

Moderation Log

New to LessWrong?

Getting Started

FAQ

Library