The Craft & The Community - A Post-Mortem & Resurrection

bybendini1y2nd Nov 2017107 comments


Epistemic status: Broad, well-developed speculation.


To my knowledge, this essay contains the most comprehensive list of criticisms of the rationality community to date. Understandably, some people may take this as a rejection of the community as a whole. It is not. In order to fix problems affecting a non-hierarchical group, individuals within the group need to have a shared understanding of them. In order to do this, someone has to look under the hood and report back with their findings.

Most people are aware there is something wrong in a general sense. There is, to some extent, an awareness that things aren’t quite right but little consensus at to whether it's analogous to a vitamin deficiency or more like a malignant neoplasm.

Solutions vary, depending on the type and extent of the problems. As such, if the consensus is that they are relatively trivial, some solutions are going to look like amputating a leg to deal with a discoloured toenail. If however, the problems are more serious, the majority of solutions proposed up to now look a lot like putting a bandaid on a bullet wound. This is the crux of it, and I expect most negative responses to this will boil down to disagreement over the size, scope and urgency of addressing the problems.

That also includes disagreements over tone; if you spot an axe murderer prowling the library, you are well within your rights to warn people by screaming at the top of your lungs. Unless you are following an absolute version of Kantian ethics, the value tradeoffs you make should vary based on what you are trying to accomplish. In the above example, alerting potential victims of an axe murderer is of far higher importance than maintaining the decorum of the library. This applies to less extreme problems too, tailored to their size, scope and urgency, while remaining mindful of the long-term consequences of bending the rules "just this once" and the pitfalls of operating in a perpetual state of emergency.

In essence, I’m taking a nuanced view of the maxim "Kind, True, Necessary". I believe every issue not omitted from the finished essay is at least relevant to talk about, every factual claim I make is to my knowledge and best efforts true, but I am not being as kind as I could possibly be. I could be kinder by writing in the abstract, but in practice that often ends up obscuring the point to such an extent that people unaware of the object level example fail to cross the inferential gap. It has often been the case in the past where I have read essays about issues within the community, assumed they were relatively trivial, then when privately informed of the details being shocked at how serious they were.

Politeness, when taken to extremes, can also have serious repercussions. If you are a meetup organizer for the kink community and you avoid telling new members there are known predators at most events, and instead cryptically suggest to "watch where you step" you are, intentionally or otherwise, prioritizing maintaining a civil atmosphere over maintaining civilization itself. Euphemizing a message to the point where those involved aren’t even aware they are being referred to runs a very high risk of missing its intended audience.

As such, this essay prioritises clarity over civility. It does not shout, nor does it speak in the gentle West Coast whisper many of you are accustomed to. Sometimes it is necessary to raise your voice slightly in order to be heard.


It has been nearly a decade since the Craft and the Community was published. Eliezer outlined a plan, hoping that someone would take the reins while he was working on AI alignment. We were told to go forth and create the art yet art creation has been overlooked, like a homeless man we try to avoid eye contact with.

Avoiding eye contact is an understandable and reasonable response for most people, given how little they can do about his situation. However it’s a little harder to justify that here, considering we still have the words "Soup Kitchen" up there in big bold letters.

Every once in awhile some wide-eyed newcomer asks why we aren’t more successful. Responses vary from minor nitpicking to claiming the rationality community is basically a bunch of people who enjoyed Yudkowsky’s blog. To me it is equal parts surreal and horrifying that people who had the reasoning ability to absorb the Sequences, including the repeated reminders that rationality is systemized winning got together with other readers and concluded that the real purpose of rationality was to have really fun conversations at dinner parties. The calm, unmoved responses make it feel like it was all some elaborate prank; that anyone who actually took the Sequences literally was either too autistic or naive to realise that they weren’t in on the joke. An implicit "Oh you sweet summer child, words don’t actually have meanings!"

This wasn’t always the consensus. Over the years there have been several dissenting voices trying to lead us in a better direction. Most of them eventually moved on when their schedules filled up or they got tired of banging their head against a wall.

One such person was Patri Friedman, known here as patrissimo. He wrote an essay on our lack of instrumental rationality and how to fix it shortly before departing. Since leaving LessWrong, his googleable accomplishments have been to get enough traction for Seasteading to sign a deal for a floating autonomous zone in French Polynesia, doing more for libertarianism than any other organization in the past few decades.

This is not a 1:1 causal relationship, but there does seem to be a correlation. To build off Patri’s analogy, there is quite the difference in outcomes between the people who stayed on the couch watching marathons, in the hopes that it would benefit their running technique when they finally got started, and those who put on whatever shoes they had to hand, got up and left the house.

Post-Mortem - What went wrong?

In retrospect, it seems somewhat surprising that a community so full of potential in both talent and values has achieved so little towards their goals when looked at as a group. There have been exceptional individuals, each with their own secret sauce they are unable to articulate the recipe for, but the median person seems to be roughly as successful as they would have been had they not discovered the rationality community. Success is not entirely, and probably not even mostly, genetic. Or in other words, information and cultural memes matter. Given this is the case, the question is why hasn’t this community managed to beat the control group?

This has been an unresolved question of mine for a few years now, and in the last several months, developed into an intense fascination.

I feel I’m starting to develop a coherent model. Like all things involving people, very little is explained by a single cause. Individual factors cluster into general areas with substantial overlaps between them. As such, looking at parts in isolation is like trying to recognise a person sitting in front of you by looking at them through a high-powered telescope. Understanding the causal mechanisms requires examination in an appropriate level of detail, being aware of individual examples and the wider context they exist in.

The following sections are an attempt to categorize them. Cleanly separating them is hard, given how interrelated they are. Yet, as with conjoined twins, you’ve got to make the incision somewhere.

The issues are separated under the broad headings of Demographics, Environment and Culture. Most issues have a cause stemming from another area and have second order effects in yet more areas still. Mapping all of these out fully would lengthen this essay to hundreds of pages, so only the most relevant connections are made.

Problems and causal factors

Demographics - Background selection effects

For various reasons, the Sequences disproportionately attracted the personality types who liked reading, hypothesising and debating. One of the defining characteristics of that personality type is a preference for extensive contemplation before action. Put enough of those people in the same place and social founder effects will exaggerate that to the point where action is rarely taken at all. From The War Of Art:

Often couples or close friends, even entire families, will enter into tacit compacts whereby each individual pledges (unconsciously) to remain mired in the same slough in which she and all her cronies have become so comfortable. The highest treason a crab can commit is to make a leap for the rim of the bucket.

Disconcerting, if true.

This is far from the only trait we have an overabundance of. Other surveys of the community suggest we have high percentages of depression, anxiety, autism and ADHD. Anecdotal evidence suggests we have a high proportion of socially maladjusted people who are some combination of heavily introverted, awkward, hyper-individualistic, oblivious, overly trusting, previously ostracised and confrontation avoidant. In an achievement sense we have a staggering percentage of people who have the intelligence to enter the higher echelons of society, but instead fell through the gaps due to burnout, untreated ADHD, major depression, defiance of authority figures, and other related causes. In a broad demographic sense, we also draw disproportionately from Blue Tribe and upper middle class backgrounds.

Even when a trait is not possessed by a majority, a tipping point mechanism can cause traits possessed in overabundance by a small minority to have wider cultural effects. It has been suggested that tipping points for opinions can happen when they are held by as little as 10% of a population.. It’s far from unreasonable to worry that a community where traits such as depression have been formally diagnosed at over twice the rate of the general population might create effects downstream.

The effects of these traits can impact our ability to achieve objectives in many cause areas. To unpack what I mean by that, here is a useful diagram borrowed from one of Raemon’s Project Hufflepuff posts:

Achieving objectives is typically thought of belonging mostly in the blue circle, but in reality, all focus areas have objectives. They may not be rooted in the standard concept of "achievement", but a sense of preferring certain outcomes/values over others.

We live in a universe where values are fragile. Where in the vast space of possible outcomes, only an infinitesimal fraction are good, and an even smaller fraction are great. Having a measurable impact is hard. Finding and preserving truth is hard. Creating flourishing communities is hard. Good outcomes do not happen by accident any more often than whirlwinds assemble Boeing 747s when passing through scrapyards.

Even when ideal, low-entropy states are achieved, constant work is required to avoid regressing to the mean. Thinking that the Rationalist community is immune to entropic forces is like thinking a refrigerator with the word cold on it will work without being plugged in.

Here are some of the problems, sorted by category, listing potential causes and secondary consequences.

Problems caused in the Truthseeking/epistemic circle:

Deep theoretical models, particularly psychological and sociological ones, don’t end up modelling reality very accurately

  • People with a high degree of social maladjustment confidently sharing faulty foundational models in group settings. Few people are present who are able to correct them, causing the "blind leading the blind" phenomenon. These theories, when uncontested, become popular, causing others to incorporate them into their theoretical models.
  • People from a small subsection of society (upper middle, high trust, blue tribe) typical minding their values, motivations and thinking processes onto people different from them.
  • A high degree of idealism preventing people from making negative adjustments to their models when the data suggests they do so.

People like Gleb Tsipursky leeching off our epistemically rigorous reputation and organizations linked to us, in order to gain status in the wider world

  • Being overly trusting, causing the principle of charity to be overapplied. This allowed the cycle of wrongdoing - accusation - response thanking critic for bringing issues to their attention - no behavioural change to repeat over long stretches of time.
  • Obliviousness to how our implicit incentive structures can be exploited for short-term gain. Gleb identified that he could spread misinformation so long as when called out, he followed the symbolic ritual of appearing contrite and agreeable with the critics, as nobody would check to ensure he had changed his behaviour. After all, no ingroup member would be so bold as to behave like it never happened.
  • Biases towards non-confrontation, which delayed public outcry once the pattern was identified.
  • People’s prior history of ostracisation for unfair reasons makes them reluctant to do something as trivial as unfriend him on Facebook. A solid third of the rationalists I’m connected to on facebook are friends with him. While friendship isn’t endorsement, it is a metric indicating your social network likes and trusts them to some degree, meaning anyone using that metric will trust Gleb more than they otherwise should.
  • Insufficient archiving of this incident, due to an underappreciation of the value in doing so. While writing this I had a hard time actually finding things I had previously read. Googling "Gleb Tsipursky lesswrong" doesn’t return any direct accounts of bad behaviour, you have to go digging for it. The right to be forgotten has its merits, but it isn’t meant to be applied when people are still doing the thing that got them in trouble in the first place.

Problems caused in the Impact/instrumental circle:

Inability to recruit underrepresented demographics

  • Group isolation has caused us to develop a distinct dialect of english that is unintelligible to people unfamiliar with it. We often try to explain things to outsiders using concepts they are unfamiliar with and circular definitions because it is the simplest way from an internal perspective; forgetting that the actual goal is to increase their understanding. The temptation to reference a concept described in the rationalist blogosphere that neatly encapsulates an idea, or drawing a comparison with something only familiar to people with backgrounds in programming, physics or science fiction is strong, and is the default choice for those who are unable to empathize with minds different from their own. This dynamic, in addition to widespread social awkwardness and a history of rejection by those different to them, creates a community that can only preach to the already converted.
  • Even if you manage to absorb the ideas, the community has only been optimized for the preferences of a narrow demographic. It has a steep dropoff in utility the more you differ from it, despite rationality as a concept having value for a much larger demographic.

Projects that aren’t run as businesses struggle

  • Cultural individualism and treating dissent as a terminal value mean most projects that run on volunteers end up being lone hero efforts. As a reaction to this, nearly all the credit ends up going to the person in charge of a project, causing contributions from the rank and file to be overlooked by external observers. People notice this and conclude there is very little reward in being part of the rank and file. This eventually bottoms out to the leader getting all of the credit, but doing all of the work minus whatever they can get from other volunteers with strong enough service impulses to override the incentive structure.
  • Analysis paralysis of individuals gets magnified in a group setting, ending up like an even worse version of this xkcd comic where it combines with short attention spans. People end up doing a less rigorous but equally unproductive version of that sort of analysis for ten minutes, then any progress on the issue is wiped from collective memory when the topic changes.
  • Project leaders often hesitate to make decisions, but non-hierarchical projects struggle to make decisions at all without some sort of leadership figure who has managed to get an implicit mandate.
  • All projects require some proportion of uninteresting, detail-oriented work. There is a finite supply of interesting/high-leverage work, which is often best done by the person in charge. This is normally solved in the business world by paying someone a share of the revenues to do the grunt work, in the form of a salary. The problem is that projects which aren’t designed to make money can’t afford to pay people for grunt work unless the leader has significant personal capital; nor does it make sense to when success means rewards are mostly distributed to the wider community rather than the leader. This often means grunt work the leader doesn’t have time for doesn’t get done at all. When this happens, the project either fails entirely or is unable to achieve some of its goals.
  • Passive volunteers who require direct, literal commands in order to get them to do anything. People for whatever reason either don’t understand, or underrate, the value of being proactive. Worse still, volunteers often don’t communicate how much time they can offer or what tasks they would be willing to do. This unintentionally causes leaders to feel like anything they ask for is draining their social capital. Efforts are often spent finding and then precisely articulating only the most interesting of tasks in the hope that they will be enticing and straightforward enough for volunteers to actually complete.
  • The number of volunteers and their enthusiasm is heavily dependant on how shiny a project feels. This means new projects with lofty goals are favoured over more established and realistic ones. Due to this incentive structure, projects must spend scarce resources on publicity that could be spent elsewhere. Projects have to repeatedly bring themselves back into the spotlight in order to replace volunteers lost to attrition. Unfortunately, all but the most pedestrian of goals require multiple years to accomplish, far longer than the average novelty-seeking rationalist takes to lose enthusiasm and move on to the next big thing.

Distrust of outsiders reducing both intake and spread of information

  • Many of the prior underachievers feel, perhaps rightly, spurned by "the system". This sometimes leads to not seeking, and often outright rejecting, assistance from groups and individuals who resemble the system that rejected them, even when it is in their interest to do so.
  • Distrust of outside experts often leads an overvaluation of skills possessed by insiders, and their expertise status is often judged more on shibboleths and willingness to frame things with insider models than actual substance. A small sample pool often means that in some deficit areas there is such a talent shortage that anyone who has even the most rudimentary level of skill is given guru status.
  • Unwillingness to learn from outside experts means we fail to avoid their mistakes. We end up reinventing the wheel, making slower advances because we are using only a narrow set of perspectives.

Lack of focus on instrumental rationality

  • Demographically we are much more naturally talented at reading, writing and debating than anything that could be described as practical. Humans generally like to do things that emphasise their strengths rather than remind them of their deficits.
  • Eliezer’s grand purpose for developing rationality was AI alignment. Groups tend to imitate their founders as it is. This was exacerbated in this case by a lack of other people pushing towards alternative causes. In the absence of sufficiently enticing competitors, AI alignment eventually sucked all the air out of the room, leaving more pedestrian causes to asphyxiate.
  • Many of our more impact-focused individuals have drifted over to the EA side of things, due to its more measurable impact and its comparatively higher status in the outside world. This lead to a productivity brain drain and a general worsening of our demographics.
  • People who are interested in individual rationality have either deemed the community a timesuck and left, or are quietly plodding away in a far corner of the blogosphere. These people stay in obscurity, as reading progress logs and life improvement tips is far less shiny than insight porn.
  • The underachieving demographic have contributed to the cultural undervaluation of hard work and attention to detail. Lack of success is partially bad genetics, but also bad cultural memes. And those bad memes end up rubbing off on the rest of the group.

Problems caused in the Human/community circle:

Romantic dissatisfaction of straight men

  • The gender gap, or more accurately, gender chasm present in the community makes it mathematically impossible for every straight man to be the primary partner of a woman within the community.
  • An inability for many straight men to communicate effectively with women not familiar with rationalism leads to them dating exclusively within the community, if at all. A lesser version of this problem exists where they struggle to communicate with women who aren’t programmers/hard science students or following similarly thing-oriented thinking patterns, all of which have similar gender disparities.
  • As many men display low levels of social skills, relationship experience, lack of self-confidence, and a vicious cycle of desperation, the few women in the community are averse to dating them. This creates a feedback loop where some men become more and more unable to get their romantic needs met.
  • The community environment is, in some aspects, passively hostile to the existence of most women, making it hard to bridge the gender gap. Hold the SJW accusations - this has very little to do with systemic misogyny, or being insufficiently charitable to feminist ideas, and much more to do with not being a place where many intellectually capable women wish to spend their time. The reasons for this are beyond the scope of a bullet point, but are expanded upon later in the essay.

Difficulty forming deeper friendships

  • Many people are socially awkward and reclusive. This goes beyond the rightly criticised introvert/extrovert binary. People who appear heavily introverted are often not as undesiring of friendships as they first appear. Towards the extreme ends of introversion, it appears to be more to do with learned helplessness than an inbuilt preference for solitude.
  • Many people simply don’t know how to make friends. Some people have a natural instinct for it, but often it’s the case that people are just doing it wrong. Forgive me for the caricature, but it seems as if people think that if they go to a few meetups, act polite, and rehash of the benefits of cryonics for five minutes with a fellow ingroup member, that they will initiate the friendship ritual, and just need to e-mail the written contract over the next morning.
  • (Cont.) Mindspace is deep and wide, it would not surprise me to learn someone had success following the above parody to the letter. But when dealing with humans, even ones who claim to be made from semiconductors, it is far from the optimal approach. Making friends requires getting to know a person, not just their topical opinions. It requires you to reveal your vulnerabilities, to spend time in each others presence. If you don’t repeatedly cross paths due to recurring shared activities, you need to be somewhat proactive, arranging to meet each other on a semi-regular basis.
  • The best methods for making friends run against the intuitions and cultural backgrounds of most people. Even if you are aware of the points mentioned previously, acting on that information is often hard when the other person lacks the context for your actions, and often ends up unintentionally sabotaging the process through a combination of social maladjustment and unexamined irrational-yet-culturally-accepted behaviours.

Difficulties executing short term plans

  • Through a combination of social anxiety, low empathy, poor time management and an inability to anticipate their future selves’ behaviour, people flake on plans they have previously agreed to. Promises are treated as expressions of enthusiasm in the present moment, not in any way binding; to be cast aside the moment they become inconvenient.
  • When people are going to be late, a lack of empathy/obliviousness sometimes means the other person isn’t even informed. Seemingly obvious boundaries are transgressed. You shouldn’t have to say "btw, if you happen to be running two hours late you should text me so that I know that."
  • A widespread inability to be where people say they are going to be, among other unreliability issues, reduces organiser morale and makes it much harder to plan events. How are you supposed to select a venue and organise logistics if you have no way of knowing whether three or twenty people are going to attend an event due to everyone making "Schrodinger plans"?

Almost complete inability to coordinate on long time horizons

  • A similar set of causal factors to the short term ones, with an (admittedly normal) inability to step outside of the short term incentive structure and take a god’s eye view of the landscape, prevents individuals from being able to coordinate on things that require a long term alignment of priorities that override short term expediency.
  • People are often taking life one day at a time, or are so enamoured with the glorious transhumanist future that if you asked what their plans are one to five years from now you will get a shrug in response. A lack of well-laid plans means people are defenceless against external economic and cultural forces. You can’t coordinate plans if you have no plans. If you go with the flow, you will go wherever the tide wishes to take you.
  • People from a Blue Tribe, individualistic middle class background have always struggled to coordinate. Tipping point effects drag the remaining minority along with them.
  • Cultural memes that make coordination easier are implicitly discouraged because they don’t match the sensibilities of the main demographic. Values like loyalty are seen as Red Tribe/outgroup traits. As such, they are usually met with some form of derision.

Lacking a sense that more is possible

  • A lot of people have not been in environments that prioritize community; as such, they have never seen glimpses of what a great community could actually look like. If all you’ve ever known is internet chatrooms and the nerd table in the school cafeteria, the current day rationalist community could easily be mistaken for peak civilization.
  • Related to that, modern atomized society is deeply flawed. Even if you agree with this premise, articulating the problems and tangible solutions is very hard. Imagining what it could look like on the outside is difficult when you have lived your entire life within its walls. Even if fish could talk, they’d have trouble articulating a life on land without