Doing good while clueless

by ioannes_shade6 min read15th Feb 201836 comments

36

Effective AltruismCause Prioritization
Frontpage

This is the fourth (and final) post in a series exploring consequentialist cluelessness and its implications for effective altruism:

  • The first post describes cluelessness & its relevance to EA; arguing that for many popular EA interventions we don’t have a clue about the intervention’s overall net impact.
  • The second post considers a potential reply to concerns about cluelessness.
  • The third post examines how tractable cluelessness is – to what extent we can grow more clueful about an intervention through intentional effort?
  • This post discusses how we might do good while being clueless to an important extent.

Consider reading the previous posts (1, 2, 3) first.


The last post looked at whether we could grow more clueful by intentional effort. It concluded that, for the foreseeable future, we will probably remain clueless about the long-run impacts of our actions to a meaningful extent, even after taking measures to improve our understanding and foresight.

Given this state of affairs, we should act cautiously when trying to do good. This post outlines a framework for doing good while being clueless, then looks at what this framework implies about current EA cause prioritization.

The following only make sense if you already believe that the far future matters a lot; this argument has been made elegantly elsewhere so we won’t rehash it here.[1]


An analogy: interstellar travel

Consider a spacecraft, journeying out into space. The occupants of the craft are searching for a star system to settle. Promising destination systems are all very far away, and the voyagers don’t have a complete map of how to get to any of them. Indeed, they know very little about the space they will travel through.

To have a good journey, the voyagers will have to successfully steer their ship (both literally & metaphorically). Let's use "steering capacity" as an umbrella term that refers to the capacity needed to have a successful journey.[2]

"Steering capacity" can be broken down into the following five attributes:[3]

  • Intent: The voyagers must have a clear idea of what they are looking for.
  • Coordination: The voyagers must be able to reach agreement about where to go.
  • Wisdom: The voyagers must be discerning enough to identify promising systems as promising, when they encounter them. Similarly, they must be discerning enough to accurately identify threats & obstacles.
  • Capability: Their craft must be powerful enough to reach the destinations they choose.
  • Predictive power: Because the voyagers travel through unmapped territory, they must be able to see far enough ahead to avoid obstacles they encounter.

This spacecraft is a useful analogy for thinking about our civilization’s trajectory. Like us, the space voyagers are somewhat clueless – they don’t know quite where they should go (though they can make guesses), and they don’t know how to get there (though they can plot a course and make adjustments along the way).

The five attributes given above – intent, coordination, wisdom, capability, and predictive power – determine how successful the space voyagers will be in arriving at a suitable destination system. These same attributes can also serve as a useful framework for considering which altruistic interventions we should prioritize, given our present situation.


The basic point

The basic point here is that interventions whose main known effects do not improve our steering capacity (i.e. our intent, wisdom, coordination, capability, and predictive power) are not as important as interventions whose main known effects do improve these attributes.

An implication of this is that interventions whose effectiveness is driven mainly by their proximate impacts are less important than interventions whose effectiveness is driven mainly by increasing our steering capacity.

This is because any action we take is going to have indirect & long-run consequences that bear on our civilization’s trajectory. Many of the long-run consequences of our actions are unknown, so the future is unpredictable. Therefore, we ought to prioritize interventions that improve the wisdom, capability, and coordination of future actors, so that they are better positioned to address future problems that we did not foresee.


What being clueless means for altruistic prioritization

I think the steering capacity framework implies a portfolio approach to doing good – simultaneously pursuing a large number of diverse hypotheses about how to do good, provided that each approach maintains reversibility.[4]

This approach is similar to the Open Philanthropy Project’s hits-based giving framework – invest in many promising initiatives with the expectation that most will fail.

Below, I look at how this framework interacts with focus areas that effective altruists are already working on. Other causes that EA has not looked into closely (e.g. improving education) may also perform well under this framework; assessing causes of this sort is beyond the scope of this essay.

My thinking here is preliminary, and very probably contains errors & oversights.


EA focus areas to prioritize

Broadly speaking, the steering capacity framework suggests prioritizing interventions that:[5]

  • Further our understanding of what matters
  • Improve governance
  • Improve prediction-making & foresight
  • Reduce existential risk
  • Increase the number of well-intentioned, highly capable people

To prioritize – better understanding what matters

Increasing our understanding of what’s worth caring about is important for clarifying our intentions about what trajectories to aim for. For many moral questions, there is already broad agreement in the EA community (e.g. the view that all currently existing human lives matter is uncontroversial within EA). On other questions, further thinking would be valuable (e.g. how best to compare human lives to the lives of animals).

Myriad thinkers have done valuable work on this question. Particularly worth mentioning is the work of the Foundational Research Institute the Global Priorities Project the Qualia Research Institute as well the Open Philanthropy Project’s work on consciousness & moral patienthood.

To prioritize – improving governance

Improving governance is largely aimed at improving coordination – our ability to mediate diverse preferences, decide on collectively held goals, and work together towards those goals.

Efficient governance institutions are robustly useful in that they keep focus oriented on solving important problems & minimize resource expenditure on zero-sum competitive signaling.

Two routes towards improved governance seem promising: (1) improving the functioning of existing institutions, and (2) experimenting with alternative institutional structures (Robin Hanson’s futarchy proposal and seasteading initiatives are examples here).

To prioritize – improving foresight

Improving foresight & prediction-making ability is important for informing our decisions. The further we can see down the path, the more information we can incorporate into our decision-making, which in turn leads to higher quality outcomes with fewer surprises.

Forecasting ability can definitely be improved from baseline, but there are probably hard limits on how far into the future we can extend our predictions while remaining believable.

Philip Tetlock’s Good Judgment Project is a promising forecasting intervention, as are prediction markets like PredictIt and polling aggregators like 538.

To prioritize – reducing existential risk

Reducing existential risk can be framed as “avoiding large obstacles that lie ahead”. Avoiding extinction and “lock-in” of suboptimal states is necessary for realizing the full potential benefit of the future.

Many initiatives are underway in the x-risk reduction cause area. Larks’ annual review of AI safety work is excellent; Open Phil has good material about projects focused on other x-risks.

To prioritize – increase the number of well-intentioned, highly capable people

Well-intentioned, highly capable people are a scarce resource, and will almost certainly continue to be highly useful going forward. Increasing the number of well-intentioned, highly capable people seems robustly good, as such people are able to diagnosis & coordinate together on future problems as they arise.

Projects like CFAR and SPARC are in this category.

In a different vein, psychedelic experiences hold promise as a treatment for treatment-resistant depression, and may also improve the intentions of highly capable people who have not reflected much about what matters (“the betterment of well people”).


EA focus areas to deprioritize, maybe

The steering capacity framework suggests deprioritizing animal welfare & global health interventions, to the extent that these interventions’ effectiveness is driven by their proximate impacts.

Under this framework, prioritizing animal welfare & global health interventions may be justified, but only on the basis of improving our intent, wisdom, coordination, capability, or predictive power.

To deprioritize, maybe – animal welfare

To the extent that animal welfare interventions expand our civilization’s moral circle they may hold promise as interventions that improve our intentions & understanding of what matters (the Sentience Institute is doing work along this line).

However, following this framework, the case for animal welfare interventions has to be made on these grounds, not on the basis of cost-effectively reducing animal suffering in the present.

This is because the animals that are helped in such interventions cannot help “steer the ship” – they cannot contribute to making sure that our civilization’s trajectory is headed in a good direction.

To deprioritize, maybe – global health

To the extent that global health interventions improve coordination, or reduce x-risk by increasing socio-political stability, they may hold promise under the steering capacity framework.

However, the case for global health interventions would have to be made on the grounds of increasing coordination, reducing x-risk, or improving another steering capacity attribute. Arguments for global health interventions on the grounds that they cost-effectively help people in the present day (without consideration of how this bears on our future trajectory) are not competitive under this framework.


Conclusion

In sum, I think the fact that we are intractably clueless implies a portfolio approach to doing good – pursuing, in parallel, a large number of diverse hypotheses about how to do good.

Interventions that improve our understanding of what matters, improve governance, improve prediction-making ability, reduce existential risk, and increase the number of well-intentioned, highly capable people are all promising. Global health & animal welfare interventions may hold promise as well, but the case for these cause areas needs to be made on the basis of improving our steering capacity, not on the basis of their proximate impacts.

Thanks to members of the Mather essay discussion group and an anonymous collaborator for thoughtful feedback on drafts of this post. Views expressed above are my own.


Footnotes

[1]: Nick Beckstead has done the best work I know of on the topic of why the far future matters. This post is a good introduction; for a more in-depth treatment see his PhD thesis, On the Overwhelming Importance of Shaping the Far Future.

[2]: I'm grateful to Ben Hoffman for discussion that fleshed out the "steering capacity" concept; see this comment thread.

[3]: Note that this list of attributes is not exhaustive & this metaphor isn't perfect. I've found the space travel metaphor useful for thinking about cause prioritization given our uncertainty about the far future, so am deploying it here.

[4]: Maintaining reversibility is important because given our cluelessness, we are unsure of the net impact of any action. When uncertain about overall impact, it’s important to be able to walk back actions that we come to view as net negative.

[5]: I'm not sure of how to prioritize these things amongst themselves. Probably improving our understanding of what matters & our predictive power are highest priority, but that's a very weakly held view.

36

36 comments, sorted by Highlighting new comments since Today at 5:35 PM
New Comment
psychedelic experiences hold promise as a treatment for treatment-resistant depression, and may also improve the intentions of highly capable people who have not reflected much about what matters (“the betterment of well people”).

I'd like to push back on this a little.

I recently did a lit review (https://srconstantin.wordpress.com/2018/02/12/atypical-treatment-resistant-depression/) on what's been shown to work for treatment-resistant depression, and while there is a study on psilocybin and it did show positive effects, it was small and uncontrolled, and fairly short-term (only three-month follow-up.)

My rough model, from anecdotal evidence, is that there is such a thing as a short-term, several-month "honeymoon" from depression that a single use of psychedelics or especially MDMA can trigger, but that these are frequently not permanent cures. We have pretty solid evidence that a single use of ketamine can produce dramatic, but short-term and rapidly-fading, reprieves from treatment-resistant depression. Sometimes temporary isn't bad -- if temporarily having your depression lifted allows you to access insights that you can carry forward into the rest of your life. But I think we should expect that one-time trips are less likely to give permanent effects than drugs that you can take every day.

The "improving the intentions of highly capable people who have not reflected much about what matters" part seems roughly true to me. I think that psychedelics are especially valuable to people who want to understand what minds are like, so, roughly, people whose work intersects with neuroscience, psychology, AI, or philosophy. I think theories about minds are bound to be incomplete if they don't include exposure to minds in very different states than the thinker's own habitual waking life. Other stuff that you should be exposed to, ideally through practical experience:

  1. How babies and young children think
  2. How animals think
  3. How mentally ill and cognitively disabled people think
  4. How people think in states of meditation or religious ecstasy
  5. How computer programs "think"

If you've ever read, say, Kant and thought "wow, has this guy literally ever played with a baby?! He's obviously wrong about what cognitive processes are universal or innate" then you have some idea of what the benefits of exposure to different minds are. Empirical experience with what minds are like seems important for "universal" questions about what is worth valuing & pursuing. Note that scientific/humanist generalists like Oliver Sacks have deep experience with most of 1-5 (I don't know that he ever studied computer science) as well as frequently having experience with psychedelics.

Thanks for the great comment :-)

I agree that the evidence around psychedelics for treatment-resistant depression is slim & preliminary. The effect sizes in Carhart-Harris et al. 2016 are very large, so I'm optimistic that a sizeable (though smaller) effect will occur in larger studies.

Sometimes temporary isn't bad -- if temporarily having your depression lifted allows you to access insights that you can carry forward into the rest of your life.

The mechanism I'm most excited about re: psychedelics for depression is a high-dose psychedelic experience functioning as a catalyst for other behavior & worldview changes which, when taken all together, lead people out of their depression.

I agree that animal welfare should be deprioritized and that understanding what matters is quite important.

The EA community's current approach to understanding what matters seems terrible to me, and has resulted in what looks to me like negative progress so far.

Consider the following analogy: suppose we live in a dystopia (hint, this isn't really an analogy) in which some vital aspect of human value has been thoroughly suppressed, say sex + romantic love to be concrete. There is no concept of sex or romantic love anywhere in the culture, except as a moral aberration that should be fought at all costs, people are taught to completely ignore their sexual or romantic urges, people get married based on genetic compatibility and reproduce via artificial insemination, etc.

Now imagine founding an organization, in this dystopia, whose mission is to try to figure out what matters by thinking, reasoning, theorizing, modeling, etc. My sense is very strongly that it will never occur to such an organization that something like sex + romantic love could be a vital aspect of human value, especially if historical mentions of other societies in which sex + romantic love mattered were either suppressed or thoroughly denigrated as morally degenerate, as long as their methodology is primarily thinking-based. If anything I expect them to come to conclusions like "well, since the strange urges we get from our genitals produce a lot of suffering in us, obviously we should be putting more funding into castration research," or whatever.

I think the only methodology that might work is for the members of such an organization to learn how to listen to their bodies, especially their sexual and romantic urges; give in to those urges despite tremendous social pressure not to and probably a bunch of what sound like excellent intellectual arguments that they shouldn't; and then decide, based on the evidence from their bodies, that what happens as a result (sex + love) feels vitally important to human value. (Of course there's the question of what methodology they could possibly follow, that would seem reasonable to them, that would bring to attention the hypothesis that they should do something like listen to their bodies; I don't have a good answer to that and that scares me.)

Edit: One general methodology that might work, although I imagine many people might find it distasteful, is to update towards past humans, which is also something that I try to do a little. Compared to the present day, this suggests that the following might be neglected components of human value: religion, violence, farming, hunting...

I worry that all of the organizations you describe as working on the problem of understanding what matters are using a thinking-based methodology that is 1) heavily influenced by cultural blindspots and 2) disconnected from the direct experience of being in a human body. The methodology I've personally been following for trying to understand what matters is to try to make my life extremely good by chasing the felt sense of "this feels vitally important to human value" through a bunch of embodied experiences, and I think I've come across important stuff that EAs don't talk about at all.

As a simple example, I don't hear people worrying a ton about the difference between just-the-head cryonics and full-body cryonics, and I personally have updated heavily in the direction of just-the-head cryonics being a terrible idea, because I think much more of what makes a person who they are is in their body and not their brain than I used to.

I find this claim very surprising - in many respects it seems in opposition to a lot of the models that are conventionally used here. (That doesn't, of course, mean that it's wrong.) I think the classic heuristics and biases view holds that many of our default intuitions are predictably inaccurate. In fact, I've often thought that this community doesn't go far enough in skepticism of default "embodied concepts" - for instance, I believe there are many both in broader society and in this community for whom human sexuality is outright negative on net, and yet I see extremely few willing to entertain this notion, even as a hypothesis.

I do think that the sort of view you describe is clearly much better than the old transhumanist "my body is a meat prison" view, but my sense was that comparatively few significantly endorsed that and little policy was based on it in any case.

I'd be interested in hearing examples of scenarios where you believe important policies seem disaligned with embodied values; I find this claim very bold and am quite interested in figuring out whether there's a good way to test it. If true, it could necessitate a serious realignment of many groups.

For instance, I think the classic heuristics and biases view holds that many of our default intuitions are predictably inaccurate.

The classic heuristics and biases literature is about things like the planning fallacy; it has very little to say about intuitions about human value, which is more in the domain of experimental moral philosophy. "Intuitions" is also a pretty broad term and arguably doesn't cleave reality at the joints; it's easy for some intuitions to fall prey to cultural blindspots (along the lines of Paul Graham's moral fashion) in exactly the way I'm worried about and I think it's an important problem to find ways of getting past blindspots in a broad sense. I trust some of my intuitions, whatever that means, much more than others for doing this.

I believe there are many both in broader society and in this community for whom human sexuality is outright negative on net, and yet I see extremely few willing to entertain this notion, even as a hypothesis.

I'm willing to entertain this as a hypothesis, although I'd be extremely sad to live in this world. I appreciate your willingness to stick up for this belief; I think this is exactly the kind of getting-past-blindspots thing we need on the meta level even if I currently disagree on the object level.

I'd be interested in hearing examples of scenarios where you believe important policies seem disaligned with embodied values; I find this claim very bold and am quite interested in figuring out whether there's a good way to test it. If true, it could necessitate a serious realignment of many groups.

So as I mentioned in another comment, I think basically all of the weird positions described in the SSC post on EAG 2017 are wrong. People who are worrying about insect suffering or particle suffering seem to me to be making philosophical mistakes and to the extent that those people are setting agendas I think they're wasting everyone's time and attention.

Personally I wonder how much of this disagreement can be attributed to prematurely settling on specifc fundamental positions or some hidden metaphysics that certain organizations have (perhaps unknowingly) committed to - such as dualism or pansychism. One of the most salient paragraphs from Scott's article said:

Morality wasn’t supposed to be like this. Most of the effective altruists I met were nonrealist utilitarians. They don’t believe in some objective moral law imposed by an outside Power. They just think that we should pursue our own human-parochial moral values effectively. If there was ever a recipe for a safe and milquetoast ethical system, that should be it. And yet once you start thinking about what morality is – really thinking, the kind where you try to use mathematical models and formal logic – it opens up into these dark eldritch vistas of infinities and contradictions. The effective altruists started out wanting to do good. And they did: whole nine-digit-sums worth of good, spreadsheets full of lives saved and diseases cured and disasters averted. But if you really want to understand what you’re doing – get past the point where you can catch falling apples, to the point where you have a complete theory of gravitation – you end up with something as remote from normal human tenderheartedness as the conference lunches were from normal human food.

I feel like this quote has some extremely deep but subtly stated insight that is in alignment with some of the points you made. Somehow, even if we all start from the position that there is no univeral or ultimately real morality, when we apply all of our theorizing, modeling, debate, measurement, thinking, etc., this somehow leads us to making absolutist conclusions about what the "truly most important thing" is. And I wonder if this is primarily a social phenomenon: In the process of debate and organizing groups of people to accomplish things, it's easier if we all converge to agreement about specific and easy to state questions.

A possible explanation for Scott's observed duality between the "suits" on the one hand who just want to do the most easily-measurable good, and the "weirdos" on the other hand who want to converge to rigorous answers on the toughest of philosophical questions (and those answers tend to look pretty bizarre), and the fact that these are often the same people - my guess is this has something to do with coverging to agreement on relatively formalizable questions. Those questions often appear in two forms: The "easy to measure" kind of questions (how many people are dying from malaria, how poor is this group of people, etc.), and the "easy to model or theorize about" questions (what do we mean by suffering, what counts as a conscious being, etc.), and so you see a divergence of activity and effort spent between those two forms of questions.

"Easy" is meant in a relative sense, of course. Unfortunately, it seems that the kind of questions that interest you (and which I agree are of crucial importance) fall into the "relatively hard" category, and therefore are much more difficult to organize concerted efforts around.

The classic heuristics and biases literature is about things like the planning fallacy; it has very little to say about intuitions about human value, which is more in the domain of experimental moral philosophy.

Fair point, though I do think it provides at least weak evidence in this domain as well. That said, there are other examples of cases where intuitions about human value can be very wrong in the moment that are perhaps more salient, - addictions and buyer's remorse come to mind.

I'm willing to entertain this as a hypothesis, although I'd be extremely sad to live in this world. I appreciate your willingness to stick up for this belief; I think this is exactly the kind of getting-past-blindspots thing we need on the meta level even if I currently disagree on the object level.

Thanks!

So as I mentioned in another comment, I think basically all of the weird positions described in the SSC post on EAG 2017 are wrong. People who are worrying about insect suffering or particle suffering seem to me to be making philosophical mistakes and to the extent that those people are setting agendas I think they're wasting everyone's time and attention.

I agree that these positions are mistakes. That said, I have three replies:

  1. I don't think the people who are making these sorts of mistakes are setting agendas or important policies. There are a few small organizations that are concerned with these matters, but they are (as far as I can tell) not taken particularly seriously aside from a small contingent of hardcore supporters.
  2. I worry that similar arguments can very easily be applied to all weird areas, even ones that may be valid. I personally think AI alignment considerations are quite significant, but I've often seen people saying things that I would parse as "being worried about AI alignment is a philosophical mistake", for instance.
  3. It is not clear to me that the "embodied" perspective you describe offers especially useful clarification on these issues. Perhaps it does in a way that I am too unskilled with this approach to understand? I (like you) think insect suffering and particle suffering are mistaken concepts and shouldn't be taken seriously, but I don't necessarily feel like I need an embodied perspective to realize that.

If I take your point very generally, I feel like you're re-stating Eliezer's Archimedes Chronophone in a cool, different way. It's saying "To discover new value, you don't just need to the same thing but rigorously - you need to be able to get outside your incentives and patterns of thought and have ideas that cause you to pull the rope sideways."

If I take your point very literally, it seems false. For example I think that trying to practically understand basic economics / game theory very rigorously gets you really far (cf. Eliezer's post "Moloch's Toolbox", Bryan Caplan's work on education and voter theory, etc). Trying to do statistics on the QALY output of differing health interventions in the developing world gets you important insights (cf. Toby Ord's "On the moral imperative toward cost-effectiveness").

(This comment is responding to your points about theory, not about the specific cognitive models being used within EA, which is a separate story.)

This didn't seem like it really addressed the point Qiaochu was making.

My guess is Qiaochu agrees that "economics and game theory can get you really far." But, would it specifically lead to EAs in the described dystopia discovering the value of sex and romance, and valuing them appropriately? I'm not sure if you're claiming it would, or that claiming that it doesn't matter especially, or something else.

I'm not sure you can pass my ITT; can you try doing that first?

Why do you think QALYs matter even approximately?

Brief response to the second question, will see if I can come back and attempt an ITT later.

QALYs don't matter much (I think), but learning that they're power-law distributed matters a lot. It's a datapoint for the general point, akin to Paul Graham's essay on black-swan farming - it tells you that value is power law distributed and that figuring out what's the most valuable thing is super-duper important, relative to how you might otherwise have been modelling the world.

Okay, let me rephrase my ITT request; what I meant to say is that I'm confused, because your chronophone analogy is pretty close to the point I'm trying to make, but I don't understand what you have in mind for my point taken "very literally" because I don't understand how your second paragraph is a response to what I wrote.

I'm pretty wary of taking this power-law methodology seriously. If you think importance is power-law distributed and your estimates of what's important are multiplicatively noisy, then the top of your list of things sorted by importance will mostly be dominated by noise. This is basically my reaction to the SSC post on EAG 2017; I think basically every weird position he describes in that post is a philosophical mistake (e.g. the wild animal suffering people seem to be using a form of total utilitarianism, which I reject completely).

First off, I'd like to apologise, I've only now read the OP, and I was talking past you. I still think we have some genuine disagreements however, so I'll try to clarify those.

I was surprised by how much I liked the post. It separates the following approaches for improving the state of the world:

  1. Further our understanding of what matters
  2. Improve governance
  3. Improve prediction-making & foresight
  4. Reduce existential risk
  5. Increase the number of well-intentioned, highly capable people

My ordering of tractability on these is roughly as follows: 5 > 4 > 3 > 2 > 1. There is then a question of importance and neglectedness. I basically think they're all fairly neglected, and I don't have strong opinions on scope except that, probably, number 4 is slightly higher than the rest.

Understanding what matters seems really hard. For the other problems (2-5) I can see strong feedback loops based in math for people to learn (e.g. in governance, people can study microeconomic models, play with them, make predictions and learn). I don't see this for problem 1.

Sure, for all 1-5 there will be steps where you have to step sideways - notice a key variable that everyone has been avoiding even thinking about, due to various incentives on what thoughts you can think - but there's more scope for practice, and for a lot of good work to be done that isn't 80% deep philosophical insight / deep rationality abilities.

Luke Muehlhauser tried really hard and found the tractability surprisingly low (1, 2).

There are worlds consistent with what I've said above, where I would nonetheless want to devote significant resources to 1, if (say) we were making plans on a 100-200 year horizon. However, I believe that we're in a world where we're very soon going to have to score perfectly on number 4, and furthermore that scoring perfectly on number 4 will cause the other problems to get much easier - including problem number 1.

Summary: As I hadn't read the OP, I read your comment as claiming your approach to 1 was the only way to do good altruistic work. I then responded with reasons to think that other - more technical approaches - were just as good (especially for the other problems). I now pivot my response to reasons to think that working on things that aren't 1 are more important, which I think we may disagree on.

I think I basically agree with all of this, except that maybe I think 1 is somewhat more tractable than you do. What I wrote was mostly a response to the OP's listing of organizations working on 1, and my sense that the OP thought that these organizations were / are making positive progress, which is far from clear to me.

I can't tell if it's still necessary, but I wanted to try anyway. Here's my ITT for your position. This story isn't literally what I think you think, but I believe it's an accurate analogy.

---

You have recently done something new, like tried LSD, or gotten really addicted to the high of exercise, or just moved to a new place and been genuinley happy and excited every minute for a whole week.

And you learned something - you didn't know life could feel this good. It's a genuine insight and feels important to you, and does change how you will make life plans forevermore.

You talk to me. I talk about the importance of figuring out what really matters in life; I've just spent many weeks reading textbooks on the subject and philosophical treatise on experience. And now you're stuck trying to communicate to me the realisation that life can be really good - you think it's causing me to make mistakes regarding all sorts of things I discuss, from population ethics to the moral value of sleep.

You know that you don't know everything there is to know about the good life, but you're pretty sure that whatever other weird realisations are out there, I'm sure not gonna find them doing what I'm doing right now.

I'm basically happy with this analogy as far as it goes, although it doesn't capture the part where part of the reason it's hard for me to communicate the thing is cultural blindspots; the dystopia analogy captures this really well and that's why I like it so much.

Regarding power laws, I am attempting to make a strong claim about how reality works - reality, not methodology. It is the case that the things we value are power law distributed - whether it's health interventions, successful startup ideas, or altruistic causes, it turns out that selecting the right one is where most of the variance is.

As a result, one's ability to do good will indeed be very noisy - this is why many good funders take hits-based approaches. For example, Peter Thiel is known for asking interesting people for their three weirdest ideas, and funding at least one of them. He funded MIRI early, though I expect he probably was quite unsure of it at the time, so I consider him to have picked up some strong hits.

That's also my feeling wrt the EA post on SSC. I'm generally not happy with the low variance approaches taken within EA and feel sad at how few new ideas are being tested, but I think that to say the number of orgs doing weird things wrt figuring out what matters, is to push the number in the wrong direction.

Sure, I'm basically happy with this modulo taking "power law" with the appropriate grains of salt (e.g. replacing with log normal or some other heavy tailed distribution as appropriate).

Have you written anything in greater detail about this (“direct experience of being in a human body”, thinking that more of what makes a person who they are is in their body, etc.)? I’d be very interested in reading it, if so!

Unfortunately no. My sense of the thing is shifting a lot and is mostly experiential and nonverbal so it feels premature and a little dangerous to try to set it down in writing too early, like I might crystallize it into the wrong form and miss the actual thing. At this point if I tried I would mostly generate poetry, some vague references to the enteric nervous system, and descriptions of experiences that might be difficult to believe along the lines of the Kensho post. I think I can transmit a sense of the thing in person but not via text.

Edit: Apparently someone wrote a book, although I haven't read it and can't vouch for it directly.

Thank you—I certainly understand your position. (And I’ll check out the link.) If you do get to a point where you can articulate this, I think it would make a great LW post (or series thereof).


This part is definitely a tangent, but here’s a follow-up question. You say:

I think I can transmit a sense of the thing in person but not via text.

I’ve heard similar things from a few folks here, lately (in the context of various discussions, on various subjects). What’s going on here? Do you have in mind that gestures would be needed to convey the thing properly, or non-verbal signals, or physical demonstrations, or… what?

So I'll start by giving what I think is an unobjectionable example of something that can't be transmitted via text, then use it to triangulate.

Consider the example of Mary's room. Mary lives in a black and white room and studies vision. (Her own body is also black and white, she has never seen her own blood, etc.) Let's stipulate that she has a perfect mechanical understanding of how human vision works, including exactly what happens when a particular wavelength of light is processed by the eye and the visual cortex as opposed to some other wavelength. It seems intuitive, and I would defend the intuition, that nonetheless Mary would learn something if she stepped out of the room and saw a red flower for the first time, with her actual eyes. Literally there are patterns of neuronal firing in her visual cortex that would happen if she did this that couldn't happen otherwise, unless someone hypnotized her into hallucinating the experience of seeing a red flower or something like that.

So, that's one example: I don't know how to transmit via text the qualia of seeing red. Worse, I don't even know how to convince Mary that there is such a thing as the qualia of seeing red, and that if she never saw red she would be missing out on something important. (I'm reminded at this point of the lack of colors in the world of The Giver, and also the SSC post on missing out on universal human experiences.)

The thing I'd want to transmit around the direct experience of being in a body has aspects of both transmitting the direct experience of seeing red and transmitting the direct experience of playing basketball, in that it has aspects of both an underutilized human capacity and learning a skill. The simplest way to do each of these is respectively to 1) show someone a red thing and 2) play basketball with them. The best you could do via text might be something like 1) give someone instructions for finding a red thing and 2) tell someone where to sign up for basketball classes, but this is not really transmitting the experience via text, it's giving someone instructions for where to find the experience themselves.

The thing I'd try to do in person, if I wanted to transmit the direct experience of being in a body, is something like run you through exercises to get you into your body, which I would ideally need to see you doing in real time to get a sense of what adjustments need to be made to the exercise. I can also demonstrate, in real time, the difference between me being in my body and not being in my body; there are things I can do in my body that are harder to do when I'm not in my body, and I have no idea how to convince you that I can do these things via text.

Via text I can at least give instructions for finding the experience, in no particular order: go to a silent meditation retreat, take psychedelic drugs, fast for 5 days and ingest nothing but water, find someone who will guide you through holotropic breathwork, find someone to perform shamanic healing on you, go to Burning Man and do as much weird shit there as you can handle. Some of these will be a lot more effective than others and I can't make guarantees about any of them. I can guide you to a basketball class but you still might just be really bad at basketball, etc.

Oh, actually, here's an exercise you can do right now, but my experience is that people vary dramatically in their ability to do it so it could easily completely fail for you, and I haven't figured out how to get it to work reliably yet. I got it from Impro. It goes like this: try to get a clear sense of where the "locus of control" of your body is. Probably it's where your brain is by default. I visualize it as something like a blue ball of energy shooting out electricity to the rest of my body to make it move. Now, try to move the locus somewhere else in your body. In particular, try to move it as far down your spine as possible; ideally you'll be able to move it to the base of your spine. For the sake of blinding I won't tell you what's supposed to happen when you do this, although there are a pretty wide range of possible responses anyway.

I tried this exercise and found it extremely interesting. My report is rot-13'ed to respect the blinding. I highly recommend taking five minutes to try it yourself before reading.

(One more line break for further encouragement to break your train of thought and try it.)

Ok, here:

V gevrq guvf fvggvat ng zl qrfx ng jbex. V'ir cerivbhfyl unq gebhoyr jvgu rkrepvfrf gung nfxrq zr gb zbir gur ybpngvba bs "zr"/zl rtb/zl crefcrpgvir bhg bs zl urnq naq vagb zl obql, fb V jnf fxrcgvpny V jbhyq fhpprrq. Ohg Dvnbpuh'f qrfpevcgvba va grezf bs "fubbgvat bhg ryrpgevpvgl gb gur erfg bs zl obql gb znxr vg zbir" znqr zr abgvpr n qvfgvapgvba orgjrra gur ybpngvba bs "gur jngpure" be zl bofreingvba/creprcgvba, naq "gur npgbe" be zl zbirzrag/npgvba. Guvf frpbaq guvat ghearq bhg gb or abg gung uneq gb zbir, juvyr gur svefg fgnlrq va cynpr.

V qba'g unir irel fgebat zragny vzntrel, fb V qvqa'g unir nalguvat yvxr n oyhr onyy bs raretl va zvaq. Zbgvba naq fcngvny eryngvbafuvc ner angheny gb zr, fb V sbphfrq ba gur vqrn bs na nofgenpg ybphf gung "chfurq bhg" gb pnhfr zbgvba.

Gb xrrc gur sbphf ba npgvba, V zbirq zl unaq va n ercrgvgvir hc-naq-qbja zbgvba juvyr V gevrq zbivat gur ybphf. Nf fbba nf vg tbg bhg bs zl urnq vagb zl hccre purfg, zl unaq zbirzragf tbg zhpu fybjre, yrff cerpvfr, naq yrff erthyne. Vg sryg n ovg yvxr jnyxvat onpxjneqf, be zbivat jvgu zl rlrf pybfrq. Gurer jnf nyfb n fbeg bs hcjneqf cerffher ba gur ybphf, yvxr jura lbh gel gb chfu fbzrguvat ohblnag qbja vagb jngre.

V ybfg genpx n pbhcyr gvzrf naq unq gb fgneg bire, ohg ba gur guveq gel V zbirq vg vagb zl thg, naq gura gb gur onfr bs zl fcvar. Nf V nccebnpurq gur onfr bs zl fcvar, na ryrpgevp, ohmmvat, fbzrjung birefgvzhyngvat raretl pragrerq gurer orpnzr fgebatre naq zber fnyvrag. Guvf raretl vf snzvyvne gb zr sebz zrqvgngvba, ohg V'ir cerivbhfyl sbhaq vg gbb vagrafr gb "ybbx ng" sbe ybat. Gb zl fhecevfr, gung jnfa'g gehr guvf gvzr, creuncf orpnhfr V jnf pbaarpgvat vg gb zbgvba naq npgvba engure guna whfg ybbxvat ng vg.

Gur raretl nebfr naq frggyrq dhvgr dhvpxyl, va n znggre bs n pbhcyr frpbaqf. Vg jnf (naq vf, nf V jevgr guvf) fgvyy cerfrag naq fnyvrag, ohg irel znantrnoyr. Nf fbba nf vg frggyrq, zl zbgvbaf jrag onpx gb orvat pbasvqrag, snfg, naq cerpvfr. Nsgre whfg n frpbaq, V (jvgubhg pbafpvbhfyl pubbfvat gb) fgbccrq gur ercrgvgvir unaq zbgvbaf naq ghearq gurz vagb n fgergpu, svefg sbe zl unaqf naq nezf, naq gura zl jubyr obql. Zl cbfgher jrag sebz fybhpurq gb fgenvtug-onpxrq (ohg ernfbanoyl erynkrq) naq V fcernq zl yrtf gb n zber pbzsbegnoyr cbfvgvba. V abgvprq gung zl cnagf jrer hapbzsbegnoyl gvtug naq erfgevpgvat, naq nyfb abgvprq gur cyrnfher bs tvivat zl obql n fngvfslvat fgergpu naq ernqwhfgzrag.

Guvf fgngr bs tbbq cbfgher naq obqvyl njnerarff vf snzvyvne gb fbzr qrterr, nf V bppnfvbanyyl trg vagb vg nsgre ubhe-ybat zrqvgngvba fvgf. Vg'f pbaarpgrq gb trareny tebhaqrqarff naq cerfrapr va fbpvny naq zragny ernyzf nf jryy, ohg vg'f n fgngr V unira'g orra noyr gb gevttre irel eryvnoyl.

V tbg hc gb tb gb nabgure ebbz, naq sbhaq zlfrys zbivat zber dhvpxyl naq pbasvqragyl guna V'z hfrq gb. Gurer'f n fjvatvat qbbe orgjrra gjb ebbzf, naq V fbeg bs yrnarq vagb vg jvgubhg oernxvat fgevqr. Zl zbgvbaf erzvaqrq zr n ovg bs ubj V'ir frra nguyrgrf va pnfhny fvghngvbaf zbir, jvgu n fbeg bs pbasvqrapr naq frafr gung gurve obql vf na rssrpgvir gbby. Vg nyfb orpnzr dhvgr boivbhf naq fnyvrag gung V'z uhatel, fvapr V unira'g rngra oernxsnfg.

V'ir xrcg gur ybphf bs npgvba ng gur onfr bs zl fcvar juvyr jevgvat guvf zrffntr. Vg frrzf gb pheeragyl erdhver n fznyy nzbhag bs onpxtebhaq nggragvba gb xrrc vg gurer (fvzvyne gb, yvxr, crepuvat ba n oenapu juvyr ernqvat n obbx, be fbzrguvat yvxr gung), ohg qbrfa'g frrz gb abgnoyl vagresrer jvgu guvaxvat be jevgvat. V srry trarenyyl sbphfrq naq ratntrq, gubhtu vg'f uneq gb gryy jurgure gung'f na rssrpg be whfg n znggre bs orvat vagrerfgrq va guvf rkcrevzrag.

Awesome! This would carry more weight if I had described in advance what some of the possible outcomes are, but this is consistent with the results I've seen and what few models I have. I'd be excited to continue working with you on this at the mentor workshop, and you can also message me on Facebook or wherever.

Thank you very much for the detailed reply!

Consider the example of Mary’s room. …

I’m quite familiar with Mary’s Room, yes, so this was indeed a helpful analogy.

I can also demonstrate, in real time, the difference between me being in my body and not being in my body; there are things I can do in my body that are harder to do when I’m not in my body, and I have no idea how to convince you that I can do these things via text.

Video!

(Also, convincing me aside, do you mind saying what these things are?)

Oh, actually, here’s an exercise you can do right now … It goes like this: try to get a clear sense of where the “locus of control” of your body is. Probably it’s where your brain is by default. I visualize it as something like a blue ball of energy shooting out electricity to the rest of my body to make it move.

I don’t think I have one of these things… (I don’t visualize any blue balls of energy, at any rate…) I suppose this is one of those “fundamental experiences that some people are missing”, etc.

Anyway, I do understand what you mean, now, and if you manage to get the phenomenon into words I will certainly read it!

Video!

I thought about it but I don't expect video to be very convincing; if I'm by myself the main differences will be changes in my body language, which I could be faking. Most of the most interesting things I can do when I'm in my body involve interacting with other people, and again on video these sorts of things could be easily faked. Try looking up videos of people being hypnotized; they're not very convincing. It's much more convincing to actually be hypnotized yourself.

(Also, convincing me aside, do you mind saying what these things are?)

I'll stick to one example: there's a thing I'm learning how to do which I'll call "talking from my guts to your guts." I'm not very good at it yet and I don't have a mechanistic explanation of how it works, but what it feels like I'm doing is an interpersonal version of Focusing. It's something like, I am trying to pick up a felt sense of what a person in front of me is feeling (e.g. from their body language, but processed mostly unconsciously), mixing that with how I'm feeling, then labeling the resulting mixed felt sense. That's not quite right, though. (Words, man.)

I've done this a few times and had it done to me a few times, and when done right I think it's a powerful tool for personal growth, although those feel like lackluster words for it.

Edit: Oops, forgot to respond to this part:

I don’t think I have one of these things… (I don’t visualize any blue balls of energy, at any rate…) I suppose this is one of those “fundamental experiences that some people are missing”, etc.

I don't mean to imply that this is something you were already doing and should be noticing that you're doing; the exercise is to start doing it. That is, start trying to conjure up the sense of a locus of control, in any way you can. I don't have a visualization that reliably works for everyone, and it's not really a visualization anyway: if visualization is something you do in your visual imagination, then the thing that's relevant here is more like doing something in your proprioceptive imagination...

The usual formulation of the Mary's Room thought experiment makes a very strong assumption about what Mary knows while in the room: that she has a complete understanding of every physical and physiological process that happens when she is looking at things. This of course goes way beyond what any real human being has.

With this assumption, I claim that Mary would already know that there are such things as the qualia of seeing red; she wouldn't need convincing. And she would already know quite a lot about those qualia; e.g., if seeing red tends to increase physiological arousal and spark thoughts of love, anger and serious physical injury then she would know that.

(This argument is not original to me; I found it in a book by Daniel Dennett.)

Without that assumption, some arguments based on the thought experiment (e.g., ones claiming that qualia have to be something nonphysical) collapse. Probably not yours, though, because for your purposes it doesn't matter whether Mary understands anything at all about colour vision; I think you get a better example of experiences that might not be conveyable in text if we suppose she doesn't.

Still... it seems like one could convey at least some aspects -- though certainly not all -- of the experience of seeing red through text, even to someone who had never had that experience. Along the following lines. "At present, your eyes give you a picture of the world that could be described by saying for everything how bright it is. Other people -- perhaps including you, some day -- get more information from their eyes than that: for each location in their visual field, not just a brightness but something else called colour. Different objects have different colours, so this lets us distinguish more things about them than you can from brightness alone. One kind of colour is called "red", and it turns out that our brains respond in interesting ways to red things. Red is the colour of a liquid called blood that we have inside us. When your body is badly enough damaged, blood may flow out. That's bad news, and seeing red makes your brain prepare a little bit to fight or run away: it gives you a little of the feeling you might have if you imagine that there's a tiger just around the corner waiting to jump on you. Most people's skin is slightly reddish in colour, especially when they are embarrassed or excited, and perhaps for that reason red is also associated in most people's minds with love and sex. Seeing red may make your brain prepare a little bit for a romantic or sexual encounter: it gives you a little of the feeling you might have if you imagine that someone you find very attractive is just around the corner waiting to kiss you. When there's something bright red in your field of view, your eyes will tend to be attracted by it, and distract your attention from other things. In our culture, red is associated with fighting and celebration and love and danger and urgency. The colour red is a bit like the sound of a trumpet fanfare, and a bit like the feel of either a warm embrace or a slap in the face."

Hearing all that (and the much more that one could say on the same theme) isn't the same as actually having the experience of seeing red, for sure. But I think someone who had never seen the colour red would know more about what seeing red is like after reading a few paragraphs of that sort of thing. They'd have some information about specific effects it might have; some analogies to other things they might be familiar with; some understanding of why those effects might be there, which might help them predict other effects; some information about ideas and feelings and images that people associate with redness.

Fair point. The version of the Mary's room setup that's relevant to this discussion is where Mary knows everything about color vision that could be transmitted via text, e.g. the best and most comprehensive possible textbook on color vision.

I agree that I could describe my experience of red to Mary, but if Mary is skeptical / doesn't trust me to report my experiences honestly I don't know how I could convince her of anything, really.

It seems you're very concerned about being thought a liar or a fake if you say more about your experiences. (E.g., you bring that up here, but so far as I can see no one in this thread has been suggesting anything of the kind.) Is there a particular reason for that?

(For the avoidance of doubt, I think it very unlikely that what you're saying -- or indeed what you're avoiding saying -- is lies or fakery.)

Mostly the response to the Kensho thread. Also some common knowledge thing along the lines of, I don't want to make strong claims on the basis of weak evidence because I don't want people to think I think other people have or should have weak epistemic standards.

I worry that all of the organizations you describe as working on the problem of understanding what matters are using a thinking-based methodology that is 1) heavily influenced by cultural blindspots and 2) disconnected from the direct experience of being in a human body.

I agree that the organizations I mentioned are all taking very cerebral approaches towards the question of what matters.

I think this is because in this essay, I was only considering the question of what current EA initiatives look promising under the steering capacity framework, and all current EA initiatives are thinking-based.

There are many other projects considering the question of what matters, many of them based in lived experience (e.g. Zen & Vipassana meditation traditions). Some of these are probably very valuable.

I really like this framing! Just a few small questions about the choice of the five attributes.

You say steering capacity has five attributes, then you make parallels to these five focus areas for altruistic work in present day work. Three q's:

  • I don't get the difference between predictive power and wisdom.
  • I would've thought predictive power would match to 'improve prediction-making & foresight', not 'number of people'.
  • Do you intend any of your points to refer to the general scientific/rationality ability of civilization, insofar as you think they're separate from prediction skills?

I don't follow your other two questions (not sure what "number of people" refers to in the second; not sure how to parse the third at all). Could you clarify or restate them?

I don't get the difference between predictive power and wisdom.

I am using "predictive power" as something like "ability to see what's coming down the pipe" and "wisdom" as something like "ability to assess whether what's coming down the pipe is good or bad, according to one's value system."

Meta: the links to previous posts seem to be broken.

Thanks – should be fixed now.