There’s an implicit model I think many people have in their heads of how everyone else behaves. As George Box is often quoted, “all models are wrong, but some are useful.” I’m going to try and make the implicit model explicit, talk a little about what it would predict, and then talk about why this model might be wrong. 

Here’s the basic idea: A person’s behavior falls on a bell curve.

I. 

Adam is a new employee at Ratburgers, a fast food joint crossed with a statistics bootcamp. It's a great company, startup investors have been going wild for machine learning in literally anything, you've been tossing paper printouts of Attention Is All You Need into the meatgrinder so you can say your hamburgers are made with AI. Anyway, you weren’t around for Adam's hiring, you haven’t seen his resume, you have no information about him when you show up to work on Monday other than it’s his first day. You walk in, see him behind the counter with a name badge, and you say hello. 

He replies with “whatever, asshole.”  

You have one datapoint about Adam, and it’s not great. If you wrote down every interaction you’d ever had with the guy, it would be a short list. It would not be a good list. Maybe you make a chart.

But work goes on, and Adam is fast and friendly when taking customer orders. He remembers everything you tell him during training, asking questions when you leave something out by accident. During lunch you give him another chance and try to strike up a conversation but he’s checked out and focused on his phone, though he’s not insulting again. By the end of the week, you have a lot more datapoints, and the asshole thing is the worst it’s been. Your mental chart looks like this:

That first interaction is still there, but you have a sense of what’s normal for Adam. If you’re aware of the Fundamental Attribution Error, you might be willing to chalk up that first morning as something unusual for him. Maybe his boyfriend broke up with him the night before, or a cat threw up in his shoes that morning. That’s not to say you don’t think it couldn’t happen again, if he has another really bad day, but it’s unusual.

There’s another way this story could have gone though. 

Maybe that first morning is pretty typical. On Tuesday he greets another employee by calling her stupid, he’s surly when customers show up, and while sometimes he’s better like when he’s doing inventory (he’s actually fairly organized) it also turns out sometimes he’s worse, like the couple of times he drops a racial epithet instead of the relatively tame “asshole,” and there was even that one time when he got into a shoving match with that one customer. Maybe the shoving match was unusually bad, but you wouldn’t be completely surprised if it happened again.

I think this is a (very simplified) way that people model each other, building up a sense of normal behavior and what the expected exceptions are. (One way it’s simplified is that I’m collapsing “good” and “bad” into one axis, where you might otherwise track “rude/polite” as one axis, “incompetent/skilled” as another, and so on.) The model gets more confident the more we’ve interacted with someone, and when we only have one or two interactions with people we often assume what we’ve seen is in the middle of the bell curve. After all, if you only get one or two samples from a population, you should expect your samples are probably close to the average.

This is one place where you and I might wind up with very different views about Adam. If you know Good Adam well, you’ve worked with him for years, you have a lot of data points suggesting that bell curve where calling someone an asshole is a rare and out of character move for him, you might be confident that nothing worse is going to happen. If I just met the guy and just have the one datapoint, I might be a lot more worried shoving people is in the distribution.

II.

On Monday Bella starts her new job at Ratburgers, a fast food joint crossed with a statistics bootcamp. You don’t know anything specific about Bella, but you do know that Ratburgers has very strict expectations on professional behavior and every employee knows about those expectations. They aren’t super high expectations, but the floor is clear and ruthlessly enforced. Wash your hands after using the bathroom, don't steal from the till, no advancing AI capabilities, and no feeding customers to the meatgrinder no matter where they stick their chewing gum.

Bella responds to your greeting the same way Adam did. “Whatever, asshole.” 

This is not against the company code of conduct, but it’s juuust on the safe side of the line. You report it anyway, and the overseer gives a mechanical “we know,” gesturing at the cameras all over the place. Over the next week, Bella continues to be surly, a demotivated worker, and even her glimmers of good behavior don’t rise to the level of great. But she never drops below the line the company will enforce. Her behavior looks like this:

Her behavior lacks any visible sign of the leftward trend that Bad Adam had. There is zero direct observation that crosses the company standard.

Pause for a moment.

III.

I’d like you to consider one of my favourite charts in the world. 

This is a beautiful chart. It is simple, illustrative, and hilarious. 

Many people will immediately spot two things wrong with the OkCupid distribution. First, it’s shifted about two inches taller. Second, there’s a totally weird lurch taller that happens right around the six foot mark. Perhaps OkCupid users are from two countries, one with better childhood nutrition, and thus this represents a population with a bimodal height distribution?

No. Approximately nobody believed that explanation for a second. What’s happening is obviously that men mostly lie (well, round up) about their height on dating apps, especially if they’re near the coveted six foot mark.

(I wonder if this lurch happens at the two meter mark in countries that use the metric system?)

It’s a subtle thing. I don’t know if I can eyeball two inches of height. But also, women’s preference for tall men is a subtle thing as well, they like lots of other things too. 

Go back to Bella’s track record. The company policy is not subtle.

Do you believe that absence of bad behavior to the left of Company Policy? Maybe. 

Especially within the company panopticon, it’s possible she’s genuinely avoiding doing anything worse. If I imagine I’m her manager I’m a little suspicious I might be missing something, but people do respond to clear rules and expectations sometimes.

Off company property and company time, or if the oversight of the supervisors has predictable blind spots, the worst behavior might be social dark matter. If I imagine being her supervisor (where the actions I take on information she’d crossed the line is predictable) and my manager asked me what I thought about promoting her, was she breaking any rules, my response might well be something like “I don’t know that she’s breaking rules, but I’m not keen to promote her.” Most organizations don’t have a flawless perfect panopticon, there’s a fair bit of fuzzy human perception going on.

Still, look at Bobbie.

I’m not particularly worried about Bobbie breaking the policy.

Or to put it another way; if you just showed me Bella’s chart and told me it was Bella’s observed behavior, I might assume that she was being worse and you just hadn’t observed it yet.

IV.

If you haven’t read Social Dark Matter, I recommend it. A lossy summary; some behaviors are punished if the behavior is spotted, and therefore people lie about or hide that behavior so you see much less of it than exists.

Some entire communities exist in the dark matter. Consider the state of cannabis users in the 1990s, or the LGBT community in 1940s USA, when getting discharged or fired or even arrested for being spotted would have been expected. And those are the ones where we now have mostly loosened up. 

Some communities exist right up against the line of The Rules of society, or even a controlled step or two past them. Take martial arts, particularly sparring focused styles. “Don’t hit people” is a basic rule of American adult life. It’s a relatively clear line, generally easy to spot visually if anyone else is around. In the dojo, there’s this narrow carve out where you can hit people in certain ways and in certain contexts. It has to be during a spar, don’t push a joint lock, if they tap out stop immediately, that kind of thing. Mistakes happen, but they happen rarely enough that overall society lets this continue.

The dojo cannot afford the kind of fuzzily perceived, long left tail distribution that Adam’s company put up with. People will get hurt. A good dojo has a few special tools for this: one of them is called honesty. If everyone is scrupulously honest about what happens, the dojo can ride closer to the line, secure in the knowledge that they will know if something goes too far.

Odd honesty norms show up in other places, and the dojo is not actually the best example. Psychonaut drug users who keep meticulous notes on the chemical compounds they imbibe, swinger and polyamorous subcultures that are shockingly upfront about what they’ve done recently that might be an STD risk, these are communities where being able to trust the people around you is a load-bearing piece of being able to do this thing safely. (While sometimes hiding themselves from society at large.)

I would be remiss if I left the rationality community out of this. Why do we have such a veneration for honesty? Yes, truth is one of our obvious core aesthetics in a way that it isn’t as obvious that psychedelic adventurers would have honesty norms. But I think there’s a dual purpose. These forums are occasionally a place to discuss topics that are generally considered taboo, or to actually ask questions that are usually a rhetorical bludgeon. “I’m not sexist and this isn’t a gotcha, but why are there fewer female computer science graduates?” is a question that people are much more willing to discuss once they’re sure you’re being honest about the ‘not sexist and this isn’t a gotcha’ part.

I would further guess that because of this common cultivation of honesty, the rationality community has an easier overlap with other groups where honesty is a load-bearing piece of being able to do a thing safely.

V.

I want to expand on that honesty bit.

Here's Carl. 

Lets say that Carl's a jerk, but Carl is a very forthright and predictable jerk. I might still feel reasonably confident that Carl isn't going to step over the line and break company policy. 

There's some specific people I've encountered who from my point of view are obnoxious, irritating, prone to taking terrible positions with little evidence to support them, and also scrupulously forthright about it. They'll come up to me unprompted and say things like "heads up, I think Xander is mad at me, probably because I said his startup idea was stupid and also he was ugly" or "In case you get complaints later, I was in a discussion about affirmative action and I think some people are upset because I told everyone immigrants have low IQ since the leaders in their home countries are evil." There's a lot wrong with this behavior! If we're arguing the line is in the wrong place and these should be over the line, sure, that's an argument I'm open to.

But I'd actually be surprised if the people I'm thinking about shoved someone. If they did, I'd genuinely expect they'd tell me they did it if I asked. I feel like I have a sense of the distribution of behavior from some of these people, and it's pretty reliable.

Now let's talk about Debbie.

It looks like Debbie is behaving pretty reasonable. There's good days and bad days, but even the bad days are comfortably short of the line. Then for one reason or another, I wind up looking closer.

This is going to be the most complicated chart I'm going to put in this essay. 

• Blue is the stuff Debbie is upfront about that actually happened.
• Purple is the stuff that did happen, but Debbie has disguised or mislead people into thinking didn't.
• Yellow is stuff that didn't happen, but Debbie has suggested or mislead people into thinking did.

Still with me?

A lot of that "average" middle ground turns out not to be real, and the tails are larger than I was aware.

If I'm Debbie's supervisor, this is a pattern I really don't like. None of what I know about is over the line. On average Debbie is still better than Carl! So why does this chart (and the people who it models) make me so concerned?

I think it's because I don't feel like I know might or might not happen. Is she going to shove a customer? Is she going to car bomb the delivery truck? I don't know. Her behavior is unpredictable, all over the place. The car bombing thing sounds unlikely, right, base rates are a thing and even if I knew nothing special about Debbie my guess should be that she isn't violent because most people aren't violent. I'm not arguing deception means horribleness.

I am arguing it can make sense to worry more how bad the behavior of people you can't trust or predict is. If Carl violated company policy I might actually think he'd tell me. If Debbie crosses that line, I expect she's going to try and cover it up the way she covered up that near violation. 

Go back to the examples above. Conscientious drug users exploring new mind-states, free love adherents enjoying each other's company, aspiring rationalists discussing taboo topics trying to discover the truth. In any of those endeavors, would you rather have Carl as a forthright jerk, or Debbie the friendly bullshitter?

And this assumes you know you've found all of the things Debbie was misleading you about. Remember Bella and her cutoff right at the policy line? With the shroud of misdirection over her actions, I might suspect that Debbie actually has been violating policy and I just don't know it. Maybe she's feeding customers to the meat grinder, or worse, giving 5% of meat grinder time to burger safety research instead of the promised 20%.

VI.

Time to anticipate some objections, and poke a few holes in this model myself.

One objection is one dimensional chart between Good and Bad is oversimplified. Plausibly there's an axis for "rude/polite" and another for "incompetent/skilled" and another for "liar/honest" and so on. I think this is true to some extent, though people do collapse these sometimes, or it gets collapsed as they talk about each other. See the Horns and Halo effects.

Some people just outright do not abide by the bell curve.

Eve  usually comes to work on time. She’s a little curt with the customer. She takes the initiative to organize inventory during slow periods. She’s willing to train new employees but she complains the entire time. As an employer (or just a fellow human being) you’re probably building a model of the kind of person she is, implicitly tracking what sorts of things she does and what would be out of character. Maybe she’ll surprise you by quitting with no notice, but probably she’s going to keep being an okay if not exemplary employee. Imagine writing down the good and bad things about her actions, adding it all up, and getting a picture something like this:

Next Monday morning, you come to work and find she’s on the news. Serial murderer, had a bunch of people buried in her basement, it’s always the quiet ones huh?

Oops.

There’s also cases where they encountered an out-of-distribution situation, and therefore behaved in a way you wouldn’t have expected. I grew up in Vermont, a state with so little racial diversity that (especially at the time I was a child) I got more negative stereotypes about Italians than I did about African Americans on account of there being were enough Italians around to have an actual out-group. I don’t think I did anything racist when I first went to university, but it’s the kind of thing where you might not have been able to generalize from my past behavior.

Maybe you just see someone at work, or around other guys, or you’ve never had to set a boundary so you’ve never seen them get mad at people who tell them no. 

You can also get people who internalize and abide by rules, not just in a “when they think someone’s looking” way. This is especially true for certain contexts or industries. I don’t know the specific rules around HIPAA or a therapist's confidentiality, but I do kind of think doctors and psychologists actually have a sharper and clearer cutoff at the line. 

(Behavioral Bell Curve dynamics seem most common when there isn't a clear cutoff and standard. My guess is the behavior of a U.S. Marine looks a lot narrower and taller.)

Also, I keep saying “bell curve” like it’s this very neat mathematical construct, but I’m basically eyeballing things. Intuitively, there’s a lot of people I feel like I have a sense of their usual behavior and how far out of it various things are and how surprised I would be if I heard they’d started a fistfight with a customer, but I don't make charts about it. 

A kind, saintly old woman who the worst you’ve ever heard of her him doing before was saying “gosh darn it” and accidentally underpaying by three cents before returning the money? I’d be very surprised. The guy with three investigations over improper behavior, whose reports say things like “While Fred’s actions were not criminal, nor technically against written policy. . .”? That’s less shocking, even if physical violence hadn’t been on his record beforehand. It's this intuitive, unspoken bell curve.

All models are wrong, but some are useful. Roll this one around in your head and see if it fits.