The Best of LessWrong

LESSWRONG
The Best of LessWrong
LW

11Daniel Kokotajlo

This is one of those posts, like "pain is not the unit of effort," that combines a memorable and informative and very useful and important slogan with a bunch of argumentation and examples to back up that slogan. I think this type of post is great for the LW review. When I first read this post, I thought it was boring and unimportant: trivially, there will be some circumstances where knowledge is the bottleneck, because for pretty much all X there will be some circumstances where X is the bottleneck. However, since then I've ended up saying the slogan "when money is abundant, knowledge is the real wealth" probably about a dozen separate times when explaining my career decisions, arguing with others at CLR about what our strategy should be, and even when deliberating to myself about what to do next. I guess longtermist EAs right now do have a surplus of money and a shortage of knowledge (relative to how much knowledge is needed to solve the problems we are trying to solve...) so in retrospect it's not surprising that this slogan was practically applicable to my life so often. I do think there are ways the post could be expanded and improved. Come to think of it, I'll make a mini-comment right here to gesture at the stuff I would add to it if I could: 1. List of other ideas for how to invest in knowledge. For example, building a community with good epistemic norms. Or paying a bunch of people to collect data / info about various world developments and report on them to you. Or paying a bunch of people to write textbooks and summaries and explainer videos and make diagrams illustrating cutting-edge knowledge (yours and others'). 2. Arguments that in fact, right now, longtermist EAs and/or AI-risk-reducers are bottlenecked on knowledge (rather than money, or power/status) --My own experience doing cost-benefit analyses is that interventions/plans vary in EV by OOMs and that it's common to find new considerations or updated models that flip the sign entirely, or ad

16Bucky

The post claims: This review aims to assess whether having read the post I can conclude the same. The review is split into 3 parts: * Epistemic spot check * Examining the argument * Outside the argument Epistemic spot check Claim: There are 14,000 nuclear warheads in the world. Assessment: True Claim: Average warhead yield <1 Mt, probably closer to 100kt Assessment: Probably true, possibly misleading. Values I found were: * US * W78 warhead: 335-350kt * W87 warhead: 300 or 475 kt * Russia * R-36 missile: 550-750 kt * R29 missile: 100 or 500kt The original claim read to me that 100kT was probably pretty close and 1Mt was a big factor of safety (~x10) but whereas the safety factor was actually less than that (~x3). However that’s the advantage of having a safety factor – even if it’s a bit misleading there still is a safety factor in the calculations. I found the lack of links slightly frustrating here – it would have been nice to see where the OP got the numbers from. Examining the argument The argument itself can be summarized as: 1. Kinetic destruction can’t be big enough 2. Radiation could theoretically be enough but in practice wouldn’t be 3. Nuclear winter not sufficient to cause extinction One assumption in the arguments for 1 & 2 is that the important factor is the average warhead yield and that e.g. a 10Mt warhead doesn’t have an outsized effect. This seems likely and a comment suggests that going over 500kt doesn’t make as much difference as might be thought and that is why warheads are the size that they are. Arguments 1 & 2 seem very solid. We have done enough tests that our understanding of kinetic destruction is likely to be fairly good so I don’t have much concerns there. Similarly, radiation is well understood and dispersal patterns seem kinda predictable in principle and even if these are wrong the total amount of radiation doesn't change, just the where it is. Climate change is less easy to model, especially giv

15DirectedEvolution

If coordination services command high wages, as John predicts, this suggests that demand is high and supply is limited. Here are some reasons why this might be true: 1. Coordination solutions scale linearly (because the problem is a general one) or exponentially (due to networking effects). 2. Coordination is difficult, unpleasant, risky work. 3. Coordination relies on further resources that are themselves in limited supply or on information that has a short life expectancy, such as involved personal relationships, technical knowhow that depends on a lot of implicit knowledge, familiarity with language and culture, access to user bases and communities, access to restricted communication channels and information, trust, credentials, charisma, money, land, or legal privileges. 4. Coordination is most intensively needed in innovative, infrastructure-development work, which is a high-risk area with long-term payoffs. 5. Coordination is neglected due to systematic biases on an individual and/or institutional level. Perhaps coordination is easy to learn, but is difficult to train in an educational context, and as such is frequently neglected by the educational system. Students are therefore mis-incentivized and don’t engage in developing their coordination skills to anywhere near the possible and optimal level. Alternatively, it might be that we teach coordination in the context of centrally coordination-focused careers (MBAs, for example), but that many other careers less obviously centrally focused on coordination (bench scientists) would also benefit - a problem of interdisciplinary neglect. Note that, if the argument in my review of interfaces as scarce resources is correct, then coordination can also be viewed as a subtype of interface - a way of translating between what a user wants and how they express that desire, into the internal language or structure of a complex system. This makes sense. Google translates natural-language queries into the PageRank algo

10DirectedEvolution

There's a lot of attention paid these days to accommodating the personal needs of students. For example, a student with PTSD may need at least one light on in the classroom at all times. Schools are starting to create mechanisms by which a student with this need can have it met more easily. Our ability to do this depends on a lot of prior work. The mental health community had to establish PTSD as a diagnosis; the school had to create a bureaucratic mechanism to normalize accommodations of this kind; and the student had to spend a significant amount of time figuring out what accommodations alleviated their PTSD symptoms and how to get them addressed through the school's bureaucracy. This points in a direction of something like "transitions research," an attempt to identify and economically address the specific barriers that skew individuals toward immediate modest-productivity strategies and away from long-term high-productivity strategies. Imagine if there was a well-known "diagnosis" of "status-loss anxiety," in which a person who's achieved some professional success notices themselves avoiding situations that would be likely to enhance their growth, yet come with a threat of loss of status. It's like the depressed person who resists mental health unseling because it implies there's something wrong with them. Being able to identify that precise reaction, label it, raise awareness of it, and find means and messages to address it would be helpful to overcome a barrier to mental health treatment. In economics jargon, what's going on here is not so much the sunk cost fallacy as a combination of aging, opportunity cost and diminishing returns. Learning takes time, aging us, and this means we have less time to profit off a new long-term investment in skill-building. Increased skill raises the opportunity cost of learning new skills. Diminishing returns means that, if we learn a skill that increases our profit from A + B to A + 2B, that this is less intrinsically valu

12DirectedEvolution

This post is based on the book Moral Mazes, which is a 1988 book describing "the way bureaucracy shapes moral consciousness" in US corporate managers. The central point is that it's possible to imagine relationship and organization structures in which unnecessarily destructive behavior, to self or others, is used as a costly signal of loyalty or status. Zvi titles the post after what he says these behaviors are trying to avoid, motive ambiguity. He doesn't label the dynamic itself, so I'll refer to it here as "disambiguating destruction" (DD). Before proceeding, I want to emphasize that DD is referring to truly pointless destruction for the exclusive purpose of signaling a specific motive, and not to an unavoidable tradeoff. This raises several questions, which the post doesn't answer. 1. Do pointlessly destructive behaviors typically succeed at reducing or eliminating motive ambiguity? 2. Do they do a better job of reducing motive ambiguity than alternatives? 3. How common is DD in particular types of institutions, such as relationships, cultures, businesses, and governments? 4. How do people manage to avoid feeling pressured into DD? 5. What exactly are the components of DD, so that we can know what to look for when deciding whether to enter into a certain organization or relationship? 6. Are there other explanations for the components of DD, and how would we distinguish between DD and other possible interpretations of the component behaviors? We might resort to a couple explanations for (4), the question of how to avoid DD. One is the conjunction of empathy and act utilitarianism. My girlfriend says she wouldn't want to go to a restaurant only she loves, even if the purpose was to show I love her. Part of her enjoyment is my enjoyment of the experience. If she loved the restaurant only she loves so much that she was desperate to go, then she could go with someone else. She finds the whole idea of destructive disambiguation of love to be distinctly unapp

33DirectedEvolution

The referenced study on group selection on insects is "Group selection among laboratory populations of Tribolium," from 1976. Studies on Slack claims that "They hoped the insects would evolve to naturally limit their family size in order to keep their subpopulation alive. Instead, the insects became cannibals: they ate other insects’ children so they could have more of their own without the total population going up." This makes it sound like cannibalism was the only population-limiting behavior the beetles evolved. According to the original study, however, the low-population condition (B populations) showed a range of population size-limiting strategies, including but not limited to higher cannibalism rates. "Some of the B populations enjoy a higher cannibalism rate than the controls while other B populations have a longer mean developmental time or a lower average fecundity relative to the controls. Unidirectional group selection for lower adult population size resulted in a multivarious response among the B populations because there are many ways to achieve low population size." Scott claims that group selection can't work to restrain boom-bust cycles (i.e. between foxes and rabbits) because "the fox population has no equivalent of the overarching genome; there is no set of rules that govern the behavior of every fox." But the empirical evidence of the insect study he cited shows that we do in fact see changes in developmental time and fecundity. After all, a species has considerable genetic overlap between individuals, even if we're not talking about heavily inbred family members, as we'd be seeing in the beetle study. Wikipedia's article on human genetic diversity cites a Nature article and says "as of 2015, the typical difference between an individual's genome and the reference genome was estimated at 20 million base pairs (or 0.6% of the total of 3.2 billion base pairs)." An explanation here is that the inbred beetles of the study are becoming progressiv

LESSWRONG
The Best of LessWrong
LW

LESSWRONG
The Best of LessWrong
LW

The Best of LessWrong

Rationality

Optimization

World

Practical

AI Strategy

Technical AI Safety