Our sensible Chesterton fences

His biased priors

Their inflexible ideological commitments

In addition to epistemic priors, there are also ontological priors and teleological priors to cross compare, each with their own problems. On top of which, people are even worse at comparing non epistemic priors than they are at comparing epistemic priors. As such, attempts to point out that these are an issue will be seen as a battle tactic: move the argument from a domain in which they have the upper hand (from their perspective) to unfamiliar territory in which you'll have an advantage, and resisted.

You may share the experience I've had that most attempts at discussion don't go anywhere. We mostly repeat our cached knowledge at each other. If two people who are earnestly trying to grok each other's positions drill down for long enough they'll get to a bit of ontology comparison, where it turns out they have different intuitions because they are using different conceptual metaphors for different moving parts of their model. But this takes so long that by the time it happens only a few bits of information get exchanged before one or both parties are too tired to continue. The workaround seems to be that if two people have a working relationship then, over time, they can accrue enough bits to get to real cruxes, and this can occasionally suggest novel research directions.

My main theory of change is therefore to find potentially productive pairings of people faster, and create the conditions under which they can speedrun getting to useful cruxes. Unfortunately, Eli Tyre tried this theory of change and reported that it mostly didn't work, after a bunch of good faith efforts from a bunch of people. I'm not sure what's next. I personally believe more progress could be made if people were trained in consciousness of abstraction (per Korzybski), but this is a sufficiently difficult ask as to fail people's priors on how much effort to spend on novel skills with unclear payoffs. And a theory of change that has a curiosity stopper that halts on "other people should do this thing that they clearly aren't going to do" is also not very useful.

Yeah, the trapped priors thing is pretty worrying to me too. But I'm confused about the opposing interventions thing. Do charter cities, or labor unions, rely on donations that much? Is it really so common for donations to cancel each other out? I guess advocacy donations (for example, pro-life vs pro-choice) do cancel each other out, so maybe we could all agree that advocacy isn't charity.

See also Rational Polarization.

I think charter cities are a questionable idea, even though I'm pro free markets. It seems that the sort of constitional change and stability required for a charter city is no easier to achieve then the kind of constitutional change and stability required for a free market in the entire country. I don't think trying either in developing countries as an outsider is a good use of anyone's resources.

[All logic is a prior.]

The anthropologist Mary Douglas covers this meta-view you have more naively described with some great biographical gaming history.

Mary Douglas argues for cultural/personal choices in which perceptions of risk (to nature, to society) inform frameworks of action/agency. I would also argue that these choices when iterated in both economic messaging (charity/consumption/display) and in conversational argument (meetings/meals/water-cooler/parliament) create the world as we know it.

https://en.wikipedia.org/wiki/Mary_Douglas

I came to her through Thought styles: Critical essays on good taste (1996) in about 2000. 

Each choice by each of us is not an aggregate in this 'structuralism', I would prefer to describe it as a pool of negotiating compositional movement at the edge of chaos & order, the big game here is complexity and survival. It's structural like a language, not like a chassis for a truck). 

It's a _lek_ we create by cooperating in order to compete on the same field (cities are mega-leks). Otherwise you will not enjoy winning starcraft, you will not enjoy complaining about what a rough deal you are getting. There is no game. There is no game in town. There is no town.

https://en.wikipedia.org/wiki/Lek_mating

Complaining is a type of (meetings/meals/water-cooler/parliament). It's part of the fun. It is where we build society, that extended phenotype that is the world. (Ideologues and narcissism always try to take the fun out of it, and make it about themselves, the only thing they can perceive. Why? because for narcs & psychopaths self=world, other people existing are a threat to that identity.

https://en.wikipedia.org/wiki/The_Extended_Phenotype

Neither the game-play nor the complaining then are dangerous or threatening, as it builds the world around us, but does requires a sense of safety in the lek. Narcissists and idealogue-emotional equivalents do not care about the safety, they feel none, and will take you down with them. (These are often called death-cults).

Further, some individuals are threatened by the lek safety itself. The safe place in which we conduct society/economy/arguments/games/marriages/buyer'sregret. While it is not always a conscious choice (they just say and do stuff to suit the moment of 'narcissistic supply') it will generally trend to push  complexity away from the stable attractors of progress and safety towards the chaos of war an coercive control.

They will do this in all organisation and labels and cultures and religions/cults. Your job is to police them in your group, the Terrans, not to point them out in the other groups. That way lies useless ineffective paranoia and conspiracy-creating conspiracy theorising (as you give in the Starcraft example). Log in your own eye.

What types of individuals? Covert narcissists (50/50 split male/female) grandiose narcissists (80/20 male/female) and the latter hold the subset of psychopaths. (All psychopaths are narcissists.)

They will engage in choices that subsumed the safety of the lek into their own "godhood". They never feel safe unless everyone else is a loser. And seen to be (as loyalist dupe or dead). They are individuals who make choices continually through the day without regard to the safety of the world/lek.

Some organisation/segments have been captured by these parasites. Which then go on to create laws which curiously favour the same behaviors. A healthy society knows how to police them.

Economics in any form, fabian, effective arseholes, marxists, weberian, austrian, hayekistians, while based on the choices made by individuals or their collectives, take no notice of the variety or complexity of psychological types, and while rational agent/actor has been criticised to death as a simplistic framework, there has been little work done to rationally exploring the diversity of human choice, it's 'structure', and why we fail to police the narcissists in our midst who are parasites on the safety of the lek created by all of us, each and collectively both. It is our job to police them.

And no it is not a witch hunt. Witches do not exist.

I kind of... hard disagree?

Effective Samaritans can't be a perfect utility inverse of Effective Altruists while keeping the labels of 'human', 'rational', or 'sane'. Socialism isn't the logical inverse of Libertarianism; both are different viewpoints on how to achieve the common goal of societal prosperity.

Effective Samaritans won't sabotage an EA social experiment any more than Effective Altruists will sabotage an Effective Samaritan social experiment. If I received a letter from Givewell thanking me for my donation that was spent on sabotaging a socialist commune, I would be very confused - that's what the CIA is for. I frankly don't expect either the next charter city or the next socialist commune to produce a flourishing society, but I do expect both to give valuable information that would allow both movements and society in general to improve their world models.

Also, our priors are not entirely trapped. It can seem that way because true political conversion rarely happens in public, and often not even consciously, but people join or leave movements regularly when their internal threshold is passed. Effective Altruist/Samaritan forums will always defend Effective Altruism/samaritanism as long as there is one EA/ES on earth, but if evidence (or social condemnation) builds up against it people whose thresholds are reached will just leave. Of course the movement as a whole can also update, but not in the same way as individual members.

People do tend to be resistant to evidence that goes against their (political) beliefs, but that resistance gets easier to surmount the fewer people there are that are less fanatical about the movement than you. Also, active rationalist practices like doubt can also help get your priors unstuck.

So in a world with EAs and ESs living side by side, there would constantly be people switching from one movement to the other or vice versa. And as either ES or EA projects get eaten by a grue, one of these rates will be greater than the other until one or both have too few supporters to do much of anything. This may have already happened (at least for the precise "Effective Samaritan" ideology of rationalism/bayesianism + socialism + individual effective giving).

So I don't think we need to resort to frequentism. Sure we can use frequentism to share the same scientific journal, but in terms of cooperation we can just all run our own experiments, share the data, and regularly doubt ourselves.

Simply run experiments and accept every result as true if the probability of it occurring by random chance falls below some threshold we agree on. This will lead us terribly astray every once in a while if we are not careful, but it also enables us to run experiments whose conclusions both of us can trust.^[2]

To minimize the chance of statistical noise or incorrect inference polluting our conclusions, we create experiments with randomly chosen intervention and control groups, so we are sure the intervention is causally connected to the outcome.

As long as we follow these procedures exactly, we can both trust the conclusion. Others can even join in on the fun too.

Together we arrive at a set of ‘randomista’ interventions we both recognize as valuable. Even if we each have differing priors leading us to opposing preferred interventions, pooling our money together on the randomista interventions beats donating to causes which cancel each other out.

The world is some the better.

Problems I see here:

1. In an RCT you can get a causal effect measure rather than just a correlation, but you can't prove whether it generalizes. Example: If you pick some people and flip a coin for each to decide whether to give them a blerg or a sblarg, and then find out that blergs has a positive effect on self-confidence, the weak spot in your inference is the way you have picked the initial set of people within whom you did the randomization. The debate moves from "you can't prove it's a causal effect!" to "your set of people where younger than normal", "the set of people is from a first-world country", "people changed too much from when you did the experiment to right now", etc.
2. Considering only significance and not power (in other words, only checking the probability something would happen "by chance", rather than the probability it would happen under specific meaningful alternative hypotheses) is limiting and can be misleading enough to be a problem in practice. Read Andrew Gelman for this stuff.
3. By Wald's complete class theorems, whatever you do with kosher frequentist statistics that you think is good, is equivalent to some specific prior used by a Bayesian guy doing Bayesian things, so if you at any point think "but frequentist" you should also have a caveat like "considering that humans won't actually use statistics kosher, I can see a pretty precise story for how doing nonsense in this specific way will produce the correct behavior in the end", which incidentally is reasoning I would not trust people in general to pull off.

Excellent post! I found the starcraft fairly amusing. Though, I am curious. Doesn't your analogy of starcraft solve the issue with trapped priors? 

Like you said, players who played all three factions mostly agree that all factions tend to be roughly similar in difficulty. However, to play all 3 factions you must arbitrarily start off playing one faction. If such people had their priors completely trapped then they wouldn't be able to change their mind after the first game, which clearly isn't true. 

I feel like even if two people disagree in theory they tend to agree in practice once you have experience with every viewpoint and point to concrete examples. (For instance, the EA and Effective Samaritan likely both agree that Denmark style social democracy is generally good while Maoist Communism is generally bad, even if they disagree on what socialism is or whether it's good or not!). 

Clearly the rationalist strategy then is not to immediately assume your right (which evidence doesn't support) but to run an experiment and figure out who's right. Notably, you shouldn't be using underhanded tactics!

I wake up to an email, thanking me and explaining how my donation has helped launch charter cities in two developing countries. Of course getting the approvals required some dirty political maneuvering, but that is the price of getting anything done.

I think of the Effective Samaritan, who has just woken up to a similar thankful email from the Developing Unions Project. In it, they explain how their donation helped make it possible for them to open a new branch of advocacy, lobbying to shut down two charter cities whose lax regulations are abused by employers to circumvent union agreements. It will require some dirty political maneuvering to get them shut down, but the ends will justify the means.

Like, this seems pretty clearly like a prisoner's dilemma doesn't it? You have concluded 'the benefits will exceed the costs' without being able to convince a reasonable opponent of the same using empirical evidence, and you went ahead and caused tangible harm anyways. Meanwhile the effective Samaritan used similar tactics to end your experiment before it bore fruit. Lose lose. You were both better off agreeing that 'underhand tactics bad' and proceeding accordingly. 

Why not just decide not to fight eachother? He starts unions in one developing country and you do a charter city in another. If one strategy is clearly better (which you both seem to insist on) then clearly the winning choice is to stop. There's no need for randomization or compromise, just moderation. You don't need to try and actively undermine eachother's efforts if we expect the results to speak for themselves. Somewhere in reality there is a truth somewhere between your worldviews. You just need to find out. 

As long as you recognize potential biases and are willing to experiment wouldn't you eventually arrive at the correct conclusions? Why bemoan the priors? They don't actually effect reality. 

There's a hint in the StarCraft description of another factor that may be at play; the environment in which people are most ideologically, culturally, socially, and intellectually aligned may be the best environment for them, ala winning more matches as their usual StarCraft faction than when switching to a perceived-stronger faction.

Similarly people's econopolitical ideologies may predict their success and satisfaction in a particular economic style because the environment is more aligned with their experience, motivation, and incentives.

If true, that would suggest that no best economical principle exists for human flourishing except, perhaps, free movement (and perhaps not free trade).

The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I don't think your dialectical reversion back to randomista logic makes much sense considering we can't exactly do random control trials to figure out any of the major questions of the social sciences. If you want to promote social science research, I think the best thing you could do is collect consistent statistics over long periods of times. You can learn a lot about modern societies just by learning how national accounts work and looking back at them many different ways. Alternatively, building agent based simulations allows you to test in flexible ways how different types of behavior, both heterogenous and homogenous, might effect macroscopic social outcomes. These are the techniques that I use and they've proven very helpful. 

If there's one other thing you're missing is this, epistemology isn't something you can rely on others for, even trying to triangulate between different viewpoints. You always have to do your own epistemology, because every way of knowing you encounter in society is a part of someone's ideological framework trying to adversarially draw you into it.