The Best of LessWrong

LESSWRONG
The Best of LessWrong
LW

24Raemon

Author here. I still endorse the post and have continued to find it pretty central to how I think about myself and nearby ecosystems. I just submitted some major edits to the post. Changes include: 1. Name change ("Robust, Coherent Agent") After much hemming and hawing and arguing, I changed the name from "Being a Robust Agent" to "Being a Robust, Coherent Agent." I'm not sure if this was the right call. It was hard to pin down exactly one "quality" that the post was aiming at. Coherence was the single word that pointed towards "what sort of agent to become." But I think "robustness" still points most clearly towards why you'd want to change. I added some clarifying remarks about that. In individual sentences I tend to refer to either "Robust Agents" or "Coherent agents" depending on what that sentence was talking about Other options include "Reflective Agent" or "Deliberate Agent." (I think once you deliberate on what sort of agent you want to be, you often become more coherent and robust, although not necessarily) Edit" Undid the name change, seemed like it was just a worse title. 2. Spelling out what exactly the strategy entails Originally the post was vaguely gesturing at an idea. It seemed good to try to pin that idea down more clearly. This does mean that, by getting "more specific" it might also be more "wrong." I've run the new draft by a few people and I'm fairly happy with the new breakdown: * Deliberate Agency * Gears Level Understanding of Yourself * Coherence and Consistency * Game Theoretic Soundness But, if people think that's carving the concept at the wrong joints, let me know. 3. "Why is this important?" Zvi's review noted that the post didn't really argue why becoming a robust agent was so important. Originally, I viewed the post as simply illustrating an idea rather than arguing for it, and... maybe that was fine. I think it would have been fine to "why" that for a followup post. But I reflected a bit on why it seemed importan

17Benquo

There are two aspects of this post worth reviewing: as an experiment in a different mode of discourse, and as a description of the procession of simulacra, a schema originally advanced by Baudrillard. As an experiment in a diffferent mode of discourse, I think this was a success on its own terms, and a challenge to the idea that we should be looking for the best blog posts rather than the behavior patterns that lead to the best overall discourse. The development of the concept occurred over email quite naturally without forceful effort. I would have written this post much later, and possibly never, had I held it to the standard of "written specifically as a blog post." I have many unfinished drafts. emails, tweets, that might have advanced the discourse had I compiled them into rough blog posts like this. The description was sufficiently clear and compelling that others, including my future self, were motivated to elaborate on it later with posts drafted as such. I and my friends have found this schema - especially as we've continued to refine it - a very helpful compression of social reality allowing us to compare different modes of speech and action. As a description of the procession of simulacra it differs from both Baudrillard's description, and from the later refinement of the schema among people using it actively to navigate the world. I think that it would be very useful to have a clear description of the updated schema from my circle somewhere to point to, and of some historical interest for this description to clearly describe deviations from Baudrillard's account. I might get around to trying to draft the former sometime, but the latter seems likely to take more time than I'm willing to spend reading and empathizing with Baudrillard. Over time it's become clear that the distinction between stages 1 and 2 is not very interesting compared with the distinction between 1&2, 3, and 4, and a mature naming convention would probably give these more natural

24LoganStrohl

* Oh man, what an interesting time to be writing this review! * I've now written second drafts of an entire sequence that more or less begins with an abridged (or re-written?) version of "Catching the Spark". The provisional title of the sequence is "Nuts and Bolts Of Naturalism". (I'm still at least a month and probably more from beginning to publish the sequence, though.) This is the post in the sequence that's given me the most trouble; I've spent a lot of the past week trying to figure out where I stand with it. * I think if I just had to answer "yes" or "no" to "do I endorse the post at this point", I'd say "yes". I continue to think it lays out a valuable process that can result in a person being much more in tune with what they actually care about, and able to see much more clearly how they're relating to a topic that they might want to investigate. * As I re-write the post for my new sequence, though, I have two main categories of objections to it, both of which seem to be results of my having rushed to publish it as a somewhat stand-alone piece so I could get funding for the rest of my work. * One category of objection I have is that it tries to do too much at once. It tries to give instructions for the procedure itself, demonstrate the procedure, and provide a grounding in the underlying philosophy/worldview. It's perhaps a noble goal to do all of that in one post, but I don't think I personally am actually capable of that, and I think I ended up falling short of my standards on all three points. If you've read my sequence Intro To Naturalism, you might possibly share my feeling that the philosophy parts of Catching the Spark are some kind of desperate and muddled. Additionally, I think the demonstration parts are insufficiently real and insufficiently diverse. When I wrote the post, I mostly looked back at my memories to find illustrative examples, rather than catching my examples in real time. A version of this with demonstrations that meet my stan

38johnswentworth

Looking back, I have quite different thoughts on this essay (and the comments) than I did when it was published. Or at least much more legible explanations; the seeds of these thoughts have been around for a while. On The Essay The basketballism analogy remains excellent. Yet searching the comments, I'm surprised that nobody ever mentioned the Fosbury Flop or the Three-Year Swim Club. In sports, from time to time somebody comes along with some crazy new technique and shatters all the records. Comparing rationality practice to sports practice, rationality has not yet had its Fosbury Flop. I think it's coming. I'd give ~60% chance that rationality will have had its first Fosbury Flop in another five years, and ~40% chance that the first Fosbury Flop of rationality is specifically a refined and better-understood version of gears-level modelling. It's the sort of thing that people already sometimes approximate by intuition or accident, but has the potential to yield much larger returns once the technique is explicitly identified and intentionally developed. Once that sort of technique is refined, the returns to studying technique become much larger. On The Comments - What Does Rationalist Self-Improvement Look Like? Scott's prototypical picture of rationalist self-improvement "starts looking a lot like therapy". A concrete image: ... and I find it striking that people mostly didn't argue with that picture, so much as argue that it's actually pretty helpful to just avoid a lot of socially-respectable stupid mistakes. I very strongly doubt that the Fosbury Flop of rationality is going to look like therapy. It's going to look like engineering. There will very likely be math. Today's "rationalist self-help" does look a lot like therapy, but it's not the thing which is going to have impressive yields from studying the techniques. On The Comments - What Benefits Should Rationalist Self-Improvement Yield? This is one question where I didn't have a clear answer

24johnswentworth

This is an excellent post, with a valuable and well-presented message. This review is going to push back a bit, talk about some ways that the post falls short, with the understanding that it's still a great post. There's this video of a toddler throwing a tantrum. Whenever the mother (holding the camera) is visible, the child rolls on the floor and loudly cries. But when the mother walks out of sight, the toddler soon stops crying, gets up, and goes in search of the mother. Once the toddler sees the mother again, it's back to rolling on the floor crying. A key piece of my model here is that the child's emotions aren't faked. I think this child really does feel overcome, when he's rolling on the floor crying. (My evidence for this is mostly based on discussing analogous experiences with adults - I know at least one person who has noticed some tantrum-like emotions just go away when there's nobody around to see them, and then come back once someone else is present.) More generally, a lot of human emotions are performative. They're emotions which some subconscious process puts on for an audience. When the audience goes away, or even just expresses sufficient disinterest, the subconscious stops expressing that emotion. In other words: ignoring these emotions is actually a pretty good way to deal with them. "Ignore the emotion" is decent first-pass advice for grown-up analogues of that toddler. In many such cases, the negative emotion will actually just go away if ignored. Now, obviously a lot of emotions don't fall into this category. The post is talking about over-applying the "ignore your emotions" heuristic, and the hazards of applying in places where it doesn't work. But what we really want is not an argument that applying the heuristic more/less often is better, but rather a useful criterion for when the "ignore your emotions" heuristic is useful. I suggest something like: will this emotion actually go away if ignored? The post is mainly talking about dealing

13DirectedEvolution

The central point of this article was that conformism was causing society to treat COVID-19 with insufficient alarm. Its goal was to give its readership social sanction and motivation to change that pattern. One of its sub-arguments was that the media was succumbing to conformity. This claim came with an implication that this post was ahead of the curve, and that it was indicative of a pattern of success among rationalists in achieving real benefits, both altruistically (in motivating positive social change) and selfishly (in finding alpha). I thought it would be useful to review 2020 COVID-19 media coverage through the month of February, up through Feb. 27th, which is when this post was published on Putanumonit. I also want to take a look at the stock market crash relative to the publication of this article. Let's start with the stock market. The S&P500 fell about 13% from its peak on Feb. 9th to the week of Feb. 23rd-Mar. 1st, which is when this article was published. Jacob sold 10% of his stocks on Feb. 17th, which was still very early in the crash. The S&P500 went on to fall a total of 32% from that Feb. 9th peak until it bottomed out on Mar. 15th. At least some gains would be made if stocks had been repurchased in the 5 months between Feb. 17th and early August 2020. I don't know how much profit Jacob realized, presuming he eventually reinvested. But this looks to me like a convincing story of Jacob finding alpha in an inefficient market, rather than stumbling into profits by accident. He didn't do it via insider knowledge or obsessive interest in some weird corner of the financial system. He did it by thinking about the basic facts of a situation that had the attention of the entire world, and being right where almost everybody else was making the wrong bet. Let's focus on the media. The top US newspapers by circulation and with a national primary service area are USA Today, the Wall Street Journal, and the New York Times. I'm going to focus on coverage in

51DirectedEvolution

1. Manioc poisoning in Africa vs. indigenous Amazonian cultures: a biological explanation? Note that while Josef Henrich, the author of TSOOS, correctly points out that cassava poisoning remains a serious public health concern in Africa, he doesn't supply any evidence that it wasn't also a public health issue in Amazonia. One author notes that "none of the disorders which have been associated with high cassava diets in Africa have been found in Tukanoans or other indigenous groups on cassava-based diets in Amazonia." Is this because Tukanoans have superior processing methods, or is it perhaps because Tukanoan metabolism has co-evolved through conventional natural selection to eliminate cyanide from the body? I don't know, but it doesn't seem impossible. 2. It's not that hard to tell that manioc causes health issues. Last year, the CDC published a report about an outbreak of cassava (manioc) poisoning including symptoms of "dizziness, vomiting, tachypnea, syncope, and tachycardia." These symptoms began to develop 4-6 hours after the meal. They reference another such outbreak from 2017. It certainly doesn't take "20 years," as Scott claims, to notice the effects. There's a difference between sweet and bitter cassava. Peeling and thorough cooking is enough for sweet cassava, while extensive treatments are needed for bitter cassava. The latter gives better protection against insects, animals, and thieves, so farmers sometimes like it better. Another analysis says that "A short soak (4 h) has no effect, but if prolonged (18 to 24 h), the amounts of cyanide can be halved or even reduced by more than six times when soaked for several days." Even if the level is cut by 1/6, is this merely slowing, or actually preventing the damage? Wikipedia says that "Spaniards in their early occupation of Caribbean islands did not want to eat cassava or maize, which they considered insubstantial, dangerous, and not nutritious." If you didn't know the difference between sweet and b

11Yoav Ravid

I remember this post very fondly. I often thought back to it and it inspired some thoughts of my own about rationality (which I had trouble writing down and are waiting in a draft to be written fully some day). I haven't used any of the phrases introduced here (Underperformance Swamp, Sinkholes of Sneer, Valley of Disintegration...), and I'm not sure whether it was the intention. The post starts with the claim that rationalists "basically got everything about COVID-19 right and did so months ahead of the majority of government officials, journalists, and supposed experts". Since it's not the point of the post I won't review this claim in depth, but it seems basically true to me. Elizabeth's review here gives a few examples. This post is about the difficulty and even danger in becoming a rationalist, or more generally, in using explicit reasoning (Intuition and Social Cognition being the alternatives). The first difficulty is that explicit reasoning alone often fails to outperform intuition and social cognition where those perform well. I think this is true, and as the rationality community evolved it came to appreciate intuition and social cognition more, without devaluing explicit reason. The second is persevering through the sneer and social pressure that comes from trying to use explicit reason to do things, often coming to very different approaches from other people, and often also failing. The third is navigating the strange status hierarchy in the community, which mostly doesn't depend on regular things like attractiveness and more often on our ability to apply explicit reason effectively, as well as being scared by strange memes like AI risk and cryonics. I don't know to what extent the first part is true in the physical communities, but it definitely is in the virtual community. The fourth is where the danger comes in. When you're in the Valley of Bad Rationality your life can get worse, and if you don't get out of it some way it might stay worse. So

19Alex_Altair

This is a negative review of an admittedly highly-rated post. The positives first; I think this post is highly reasonable and well written. I'm glad that it exists and think it contributes to the intellectual conversation in rationality. The examples help the reader reason better, and it contains many pieces of advice that I endorse. But overall, 1) I ultimately disagree with its main point, and 2) it's way too strong/absolutist about it. Throughout my life of attempting to have true beliefs and take effective actions, I have quite strongly learned some distinction that maps onto the ideas of inside and outside view. I find this distinction extremely helpful, and specifically, remembering to use (what I call) the outside view often wins me a lot of Bayes points. When I read through the Big Lists O' Things, I have these responses; * I think many of those things are simply valid uses of the terms[1] * People using a term wrong isn't a great reason[2] to taboo that term; e.g. there are countless mis-uses of the concept of "truth" or "entropy" or "capitalism", but the concepts still carve reality * Seems like maybe some of these you heard one person use once, and then it got to go on the list? A key example of the absolutism comes from the intro: "I recommend we permanently taboo “Outside view,” i.e. stop using the word and use more precise, less confused concepts instead." (emphasis added). But, as described in the original linked sequence post, the purpose of tabooing a word is to remember why you formed a concept in the first place, and see if that break-down helps you reason further. The point is not to stop using a word. I think the absolutism has caused this post to have negative effects; the phrase "taboo the outside view" has stuck around as a meme, and in my memory, when people use it it has not tended to be good for the conversation. Instead, I think the post should have said the following. * The term "outside view" can mean many things that can

12Raemon

This is the post that first spelled out how Simulacra levels worked in a way that seemed fully comprehensive, which I understood. I really like the different archetypes (i.e. Oracle, Trickster, Sage, Lawyer, etc). They showcased how the different levels blend together, while still having distinct properties that made sense to reason about separately. Each archetype felt very natural to me, like I could imagine people operating in that way. The description Level 4 here still feels a bit inarticulate/confused. This post is mostly compatible with the 2x2 grid version, but it makes the additional claim that Level 4 don't know how to make plans, and are 'particularly hard to grok.' It bundles in some worldview from Immoral Mazes / Raoian Sociopaths. For me, a big outstanding question re: Simulacra is "does it actually make sense to bundle the Kafkaesque sociopath who can't make plans as an explicit part of Level 4?" I think this is a kinda empirical question. An example of the sort of evidence that'd persuade me are "among politicians or middle managers who spend most of their time optimizing for power, interacting with facts and tribal affiliations as a game, what proportion of them actually lose their ability to make plans, or otherwise become more... lovecraftian or whatever?" Is it more like "70%", "50%", "10%"?. It's plausible to me that there's a relatively small number of actors who stand out as particularly extreme (and then get focused on for toxoplasma of rage reasons) Or, rather: if I simply describe Primarily Level 4 people as "holding social-signaling as object", am I actually missing anything? Do they tend to have any attributes? What? ... I do this post is among the best intro to the Simulacra Levels concept, and think it's worth polishing up slightly. I assume Zvi has thought a bit more about Level 4 by now. If it still seems like there's something Importantly, Confusingly Up With Them, I'm hoping that can be spelled out a bit more. (I think my fav

18Screwtape

I think this, or something like this, should be in a place of prominence on LessWrong. The Best Of collection might not be the place, but it's the place I can vote on, so I'd like to vote for it here. I used "or something like this" above intentionally. The format of this post — an introduction of why these guidelines exist, short one or two sentence explanations of the guideline, and then expanded explanations with "ways you might feel when you're about to break the X Guideline" — is excellent. It turns each guideline into a mini-lesson, which can be broken out and referenced independently. The introduction gives context for them all to hang together. The format is A+, fighting for S tier. Why "something like this" instead of "this, exactly this" then? Each individual guideline is good, but they don't feel like they're the only set. I can imagine swapping basically any of them other than 0 and 1 out for something different and having something I liked just as much. I still look at 5 ("Aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth") and internally wince. I imagine lots of people read it, mostly agreed with it, but wanted to replace or quibble with one or two of the guidelines, and from reading the comments there wasn't a consensus on which line was out of place. That seems like a good sign. It's interesting to me to contrast it with Elements Of Rationalist Discourse. Elements doesn't resonate as much with me, and while some of that is Elements is not laid out as cleanly I also don't agree with the list the same way. And yet, Elements was also upvoted highly. The people yearn for guidelines, and there wasn't a clear favourite. Someday I might try my own hand at the genre, and I still consider myself to owe an expansion on my issues with 5. I'm voting for this to be in the Best Of LessWrong collection. If there was a process to vote to make this or at least the introduction and Guidelines, In Brief in

19Raemon

Self Review. I still endorse the broad thrusts of this post. But I think it should change at least somewhat. I'm not sure how extensively, but here are some considerations Clearer distinctions between Prisoner's Dilemma and Stag Hunts I should be more clear about what the game theoretical distinctions I'm actually making between Prisoners Dilemma and Stag Hunt. I think Rob Bensinger rightly criticized the current wording, which equivocates between "stag hunting is meaningfully different" and "'hunting rabbit' has nicer aesthetic properties than 'defect'". I think Turntrout spelled out in the comments why it's meaningful to think in terms of stag hunts. I'm not sure it's the post's job to lay it out in the exhaustive detail that his comment does, but it should at least gesture at the idea. Future Work: Explore a lot of coordination failures and figure out what the actual most common rules / payoff structures are. Stag Hunting is relevant sometimes, but not always. I think it's probably more relevant than Prisoner's Dilemma, which is a step up, but I think it's worth actually checking which game theory archetypes are most relevant most of the time. Reworked Example Some people comment that my proposed stag hunt... wasn't a stag hunt. I think that's actually kind of the point (i.e. most things that look like stag hunts are more complicated than you think, and people may not agree on the utility payoff). Coming up with good examples is hard, but I think at the very least the post should make it more clear that no, my original intended Stag Hunt did not have the appropriate payoff matrix after all. What's the correct title? While I endorse most of the models and gears in this post, I... have mixed feelings about the title. I'm not actually sure what the key takeaway of the post is meant to be. Abram's comment gets at some of the issues here. Benquo also notes that we do have plenty of stag hunts where the schelling choice is Stag (i.e. don't murder) I think

17Zvi

This is a long and good post with a title and early framing advertising a shorter and better post that does not fully exist, but would be great if it did. The actual post here is something more like "CFAR and the Quest to Change Core Beliefs While Staying Sane." The basic problem is that people by default have belief systems that allow them to operate normally in everyday life, and that protect them against weird beliefs and absurd actions, especially ones that would extract a lot of resources in ways that don't clearly pay off. And they similarly protect those belief systems in order to protect that ability to operate in everyday life, and to protect their social relationships, and their ability to be happy and get out of bed and care about their friends and so on. A bunch of these defenses are anti-epistemic, or can function that way in many contexts, and stand in the way of big changes in life (change jobs, relationships, religions, friend groups, goals, etc etc). The hard problem CFAR is largely trying to solve in this telling, and that the sequences try to solve in this telling, is to disable such systems enough to allow good things, without also allowing bad things, or to find ways to cope with the subsequent bad things slash disruptions. When you free people to be shaken out of their default systems, they tend to go to various extremes that are unhealthy for them, like optimizing narrowly for one goal instead of many goals, or having trouble spending resources (including time) on themselves at all, or being in the moment and living life, And That's Terrible because it doesn't actually lead to better larger outcomes in addition to making those people worse off themselves. These are good things that need to be discussed more, but the title and introduction promise something I find even more interesting. In that taxonomy, the key difference is that there are games one can play, things one can be optimizing for or responding to, incentives one can creat

38Ben Pace

Here are my thoughts. 1. Being honest is hard, and there are many difficult and surprising edge-cases, including things like context failures, negotiating with powerful institutions, politicised narratives, and compute limitations. 2. On top of the rule of trying very hard to be honest, Eliezer's post offers an additional general rule for navigating the edge cases. The rule is that when you’re having a general conversation all about the sorts of situations you would and wouldn’t lie, you must be absolutely honest. You can explicitly not answer certain questions if it seems necessary, but you must never lie. 3. I think this rule is a good extension of the general principle of honesty, and appreciate Eliezer's theoretical arguments for why this rule is necessary. 4. Eliezer’s post introduces some new terminology for discussions of honesty - in particular, the term 'meta-honesty' as the rule instead of 'honesty'. 5. If the term 'meta-honesty' is common knowledge but the implementation details aren't, and if people try to use it, then they will perceive a large number of norm violations that are actually linguistic confusions. Linguistic confusions are not strongly negative in most fields, merely a nuisance, but in discussions of norm-violation (e.g. a court of law) they have grave consequences, and you shouldn't try to build communal norms on such shaky foundations. 6. I and many other people this post was directed at, find it requires multiple readings to understand, so I think that if everyone reads this post, it will not be remotely sufficient for making the implementation details common knowledge, even if the term can become that. 7. In general, I think that everyone should make sure it is acceptable, when asking "Can we operate under the norms of meta-honesty?" for the other person to reply "I'd like to taboo the term 'meta-honesty', because I'm not sure we'll be talking about the same thing if we use that term." 8. This is a valuable bedrock for thinking

13Screwtape

The thing I want most from LessWrong and the Rationality Community writ large is the martial art of rationality. That was the Sequences post that hooked me, that is the thing I personally want to find if it exists. Therefore, posts that are actually trying to build a real art of rationality (or warn of failed approaches) are the kind of thing I'm going to pay attention to, and if they look like they actually might work I'm going to strongly vote for including them in the Best Of LessWrong collection. Feedbackloop-first Rationality sure looks like an actual attempt at solving the problem. It lays out a strategy, the plan seems like it plausibly might work, and there's followup workshops that suggest some people are actually willing to spend money on this; that's not a clear indicator that it works (people spend money on all kinds of things) but it is significantly more than armchair theorizing. If Raemon keeps working on this and is successful, I expect we'll see some testable results. If, say, the graduates or regular practitioners turn out to be able to confidently one-shot Thinking Physics style problems while demographically matched people stumble around, that'll be a Hot Dang Look At That Chart result at least in the toy problems. If they go on to solve novel, real world problems, then that's a clear suggestion this works. There's two branches of followup I'd like to see. One, Raemon's already been doing; running more workshops teaching this, teasing out useful subskills to teach, and writing up how how to run exercises and what the subskills are. The second is evaluations. If Raemon's keeping track of students and people who considered going but didn't, I'd love to see a report on how both sets are doing in a year or two. I'm also tempted to ask on future community censuses whether people have done Feedbackloop-first Rationality workshops (["Yes under Raemon", "Yes by other people based on this", "no"] and then throw a timed Thinking Physics-style problem a

17DirectedEvolution

The goal of this post is to help us understand the similarities and differences between several different games, and to improve our intuitions about which game is the right default assumption when modeling real-world outcomes. My main objective with this review is to check the game theoretic claims, identify the points at which this post makes empirical assertions, and see if there are any worrisome oversights or gaps. Most of my fact-checking will just be resorting to Wikipedia. Let’s start with definitions of two key concepts. Pareto-optimal: One dimension cannot improve without a second worsening. Nash equilibrium: No player can do better by unilaterally changing their strategy. Here’s the payoff matrix from the one-shot Prisoner’s Dilemma and how it relates to these key concepts. B stays silentB betraysA stays silentPareto-optimal A betrays Nash equilibrium This article outlines three possible relationships between Pareto-optimality and Nash equilibrium. 1. There are no Pareto-optimal Nash equilibria. 2. There is a single Pareto-optimal Nash equilibrium, and another equilibrium that is not Pareto-optimal. 3. There are multiple Pareto-optimal Nash equilibria, which benefit different players to different extents. The author attempts to argue which of these arrangements best describes the world we live in, and makes the best default assumption when interpreting real-world situations as games. The claim is that real-world situations most often resemble iterated PDs, which have multiple Pareto-optimal Nash equilibria benefitting different players to different extents. I will attempt to show that the author’s conclusion only applies when modeling superrational entities, or entities with an unbounded lifespan, and give some examples where this might be relevant. Iterated Prisoner’s Dilemma is a little more complex than the author states. If the players know how many turns the game will be played for, or if the game has a known upper limit of t

24Bucky

A short note to start the review that the author isn’t happy with how it is communicated. I agree it could be clearer and this is the reason I’m scoring this 4 instead of 9. The actual content seems very useful to me. AllAmericanBreakfast has already reviewed this from a theoretical point of view but I wanted to look at it from a practical standpoint. *** To test whether the conclusions of this post were true in practice I decided to take 5 examples from the Wikipedia page on the Prisoner’s dilemma and see if they were better modeled by Stag Hunt or Schelling Pub: * Climate negotiations * Relationships * Marketing * Doping in sport * Cold war nuclear arms race Detailed analysis of each is at the bottom of the review. Of these 5, 3 (Climate, Relationships, Arms race) seem to me to be very well modeled by Schelling Pub. Due to the constraints on communication allowed between rival companies it is difficult to see marketing (where more advertising = defect) as a Schelling Pub game. There probably is an underlying structure which looks a bit like Schelling Pub but it is very hard to move between Nash Equilibria. As a result I would say that Prisoner’s Dilemma is a more natural model for marketing. The choice of whether to dope in sport is probably best modeled as a Prisoner’s dilemma with an enforcing authority which punishes defection. As a result, I don’t think any of the 3 games are a particularly good model for any individual’s choice. However, negotiations on setting up the enforcing authority and the rules under which it operates are more like Schelling Pub. Originally I thought this should maybe count as half a point for the post but thinking about it further I would say this is actually a very strong example of what the post is talking about – if your individual choice looks like a Prisoner’s Dilemma then look for ways to make it into a Schelling Pub. If this involves setting up a central enforcement agency then negotiate to make that happen. So I

19fiddler

This post seems excellent overall, and makes several arguments that I think represent the best of LessWrong self-reflection about rationality. It also spurred an interesting ongoing conversation about what integrity means, and how it interacts with updating. The first part of the post is dedicated to discussions of misaligned incentives, and makes the claim that poorly aligned incentives are primarily to blame for irrational or incorrect decisions. I’m a little bit confused about this, specifically that nobody has pointed out the obvious corollary: the people in a vacuum, and especially people with well-aligned incentive structures, are broadly capable of making correct decisions. This seems to me like a highly controversial statement that makes the first part of the post suspicious, because it treads on the edge of proving (hypothesizing?) too much: it seems like a very ambitious statement worthy of further interrogation that people’s success at rationality is primarily about incentive structures, because that assumes a model in which humans are capable and preform high levels of rationality regularly. However, I can’t think of an obvious counterexample (a situation in which humans are predictably irrational despite having well-aligned incentives for rationality), and the formulation of this post has a ring of truth for me, which suggests to me that there’s at least something here. Conditional on this being correct, and there not being obvious counterexamples, this seems like a huge reframing that makes a nontrivial amount of the rationality community’s recent work inefficient-if humans are truly capable of behaving predictably rationally through good incentive structures, then CFAR, etc. should be working on imposing external incentive structures that reward accurate modeling, not rationality as a skill. The post obliquely mentions this through discussion of philosopher-kings, but I think this is a case in which an apparently weaker version of a thesis actually i

LESSWRONG
The Best of LessWrong
LW

LESSWRONG
The Best of LessWrong
LW

The Best of LessWrong

Rationality

Optimization

World

Practical

AI Strategy

Technical AI Safety