LESSWRONG
LW

4068
Haiku
3340710
Message
Dialogue
Subscribe

I am a volunteer organizer with PauseAI and PauseAI US, a top forecaster, and many other things that are currently much less important.

The risk of human extinction from artificial intelligence is a near-term threat. Time is short, p(doom) is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes.

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
No posts to display.
No wikitag contributions to display.
Tomás B.'s Shortform
Haiku3d30

I have definitely multiple times had the thought, "I don't have a problem with X!" Only to later realize that I was subconsciously structuring my life so that I would never encounter X. That means if I do encounter X, I might be unprepared and have a bad time.

I think this is very common. It's like not noticing you have a hurt muscle, by subconsciously compensating with the other ones. That's fine until you end up painfully contorted and some muscle somewhere finally sends a strong signal to make you unambiguously aware of the situation.

Reply
At odds with the unavoidable meta-message
Haiku3d94

I once had a codependent friendship in which there was a pressure to keep messages short, speak in reference and metaphor, and not explain at length what I meant. For this and other reasons, that relationship was toxic and frequently emotionally painful.

After that friendship ended, I read only the first few chapters of the excellent book Codependent No More, and I was abruptly and permanently cured. That isn't how that normally works; it's usually "once codependent, always codependent," like alcoholism. But it turned out my problem was fundamentally a lack of knowledge and understanding. I straight-up did not know that I was not responsible for other people's feelings. Gaining that understanding was curative.

This is relevant to @Gordon Seidoh Worley's and @Raemon's conversation. It is good to be thoughtful. It is bad to take upon yourself the full weight of other people's response to you. Do be considerate. Don't manage other people's feelings. I'm not good at articulating the difference, but that book is the ultimate work on the subject.

The other message I want to convey here is on the opposite, positive end: It is good to get practiced at tactfully and efficiently explaining yourself, asking for clarification, expressing vulnerability and anxiety about a communication, etc., all with appropriate proportionality, and especially with people you are close to. Bad communication is usually fixed by subsequent good communication, and rarely so by less communication.

My partner and I have intentionally cultivated space for a rich and easily accessible meta-conversation, which we can jump into at any time with cues like "Put what I'm about to say in a bubble," (i.e. "I have a potentially dangerous thought that I'm not sure I fully endorse, but it feels like there's something important in there that I ought to share, and I am asking not to be judged for it or for it to go onto the permanent record while I work it out with you in real time") or "Can I ask an anxiety question?" (i.e. "I have social anxiety about something between us, and I would like to express it without implying that you have done something wrong to cause this anxiety"). Other than love (mutual, deep, genuine good intention toward each other), I believe the meta-conversation is the single most impactful innovation that has allowed our relationship to be abnormally healthy over the long run.

Reply
Richard_Kennaway's Shortform
Haiku3d10

The little girl in danger of drowning before your eyes is worth exactly the same as every child everywhere in danger of death. To fail to do all you can to save all the people you can is morally equivalent to passing by the drowning girl.

The former does not imply the latter. That's my point here, is that despite their lives having equal worth, the situations are not equivalent. Likewise, the "large man" variation of the trolley problem is not an equivalent scenario to the original. If the situations were equivalent, they would have equivalent effects. But they do not have equivalent effects. The alternative choices are not felt the same way, and they are not perceived the same way. They do not affect the actors or onlookers the same way, therefore the effect is not the same. The effects are not equivalent, meaning the outcome is not equivalent, so the choices can't be equivalent.

For a taste of this: Wouldn't walking past the drowning girl harden your heart against all manner of clear and present suffering, and cause distant onlookers to condemn you, perhaps permanently ruining your reputation? Wouldn't pushing a man off a bridge land you in prison and give you nightmares for the rest of your life, or make you more willing to take a life in the future for lesser reasons?

Any argument that these scenarios should be equivalent is analogous to a physics student proclaiming that cows "ought to be" spherical. When we do physics, we strip out a tremendous amount of detail to simplify our calculations. We turn it into math. If our assumptions are correct, the loss of detail won't significantly change the final result, and we can turn the math back into a prediction about physics.

From the standpoint of consequentialism, any moral philosophy that creates these equivalencies is incorrect, in an objective, predictive sense. Humans are not spherical cows in a moral vacuum. When a scenario is presented that adds the atmosphere back in, we do not get to continue with our simplified assumptions without consequence. Human psychology is a very important part of human morality, and ignoring its role doesn't just produce counter-intuitive results, it produces bad predictions, which if acted upon can have devastating consequences.

In the scenario presented, anyone choosing not to save the drowning girl in a real life version of this situation would be a deeply rotten person who I would not feel safe being around, and that feeling would correspond to a calibrated prediction about the type of behavior I should expect them to engage in in the future.

The scenario is not necessary to argue that it is good to do good in the world that you cannot see. But it can be used to justify doing evil.

Reply
Open Thread Autumn 2025
Haiku8d30

Surely you're not really making any decisions.

Something is making decisions, is it not? And that thing that makes the decisions is part of what you would normally describe as "you." Everything still adds up to normality.

It can can be detrimental, though, to communicate certain subsets of true things without additional context, or in a way that is likely to be misinterpreted by the audience. Communicating truth (or at least not lying) is more about the content that actually ends up in people's heads than it is about the content of the communication itself.

Reply
Richard_Kennaway's Shortform
Haiku8d10

I find myself somewhat disappointed that the scenario and the response don't seem to have much to say. I actually thought the original scenario was meant as a thinly-veiled scathing critique of naive utilitarianism. For any non-psychopath who is actually in the scenario, saving the drowning girl doesn't require any moral argumentation, and I wouldn't feel safe being in community with anyone who sincerely preferred the other ending.

I think when people first discover rationalism and effective altruism, they often descend into a valley of learning to "shut up and multiply" and casting their intuitions aside. It's very dangerous to remain in that valley without gaining humility about the limits of those tools. To be one of the good guys, you do have to rejoin human society and continue to love people in normal ways, and that isn't only about one's own personal warm fuzzies.

The neutrality of your follow-up reveals the original scenario as something like a scaffold upon which to hang one's perspective and observe other perspectives being hung. The original example leaves me with a lingering itch, though, because the alternate perspective is not something that anyone should be encouraged to entertain.

Reply
Richard_Kennaway's Shortform
Haiku11d31

I thought another one: 4. Beware of simplifying assumptions and the Either/Or fallacy. You do not live in a moral philosophy dilemma. Reality has high dimensionality and the trade-offs are not always obvious and clean.

Reply
Richard_Kennaway's Shortform
Haiku11d32
  1. Second-order effects matter. Doing X is not only doing X. Becoming the type of person who does things in the reference class of X is one of the effects of your actions. Your reputation can likewise be affected. Your behavior can be copied by others, sometimes lossily. If it would be downright catastrophic for a chunk of society to do things vaguely shaped like X on a regular basis, you probably shouldn't do X.
  2. If a utilitarian calculus concludes that you should do something that strongly violates societal norms of morality (a la virtue effects and deontology), your math is probably bad. Be humble. (You probably didn't invent new physics while doing your math homework, you probably didn't discover that "letting kids drown is sometimes good actually.")
  3. Be suspicious of unintuitive moral conclusions that violate norms in your favor.
Reply
No, That's Not What the Flight Costs
Haiku11d30

I'm curious about this as well, since as far as I'm aware, flights in the US tend to be more expensive than flights in, say, Europe.

Reply
Benito's Shortform Feed
Haiku15d85

I'm very happy with most of these choices. I'm confused about "locally valid" and "locally invalid" being removed, though. It would feel weird to me to use "weak argument" and "strong argument" in their stead, since weak and strong connote doing different amounts of work, while valid and invalid denote logical applicability.

Reply
"Shut It Down" is simpler than "Controlled Takeoff"
Haiku19d512

Thank you for your high-quality engagement on this and for including the clear statement!

I think my most substantial disagreement with you on the difficulty of a shutdown is related to longtermism. Most normal people would not take a 5% risk of destroying the world in order to greatly improve their lives and the lives of their children. That isn't because they are longtermist, but primarily because they are simply horrified by the concept of destroying the world.

It is in fact almost entirely utilitarians who are in favor of taking that risk, because they are able to justify it to themselves after doing some simplified calculation. Ordinary people, rational or irrational, who just want good things for themselves and their kids usually don't want to risk their own lives, certainly don't want to risk their kids' lives, and it wouldn't cross their mind to risk other people's kids' lives, when put in stark terms.

"Human civilization should not be made to collapse in the next few decades" and "humanity should survive for a good long while" are longtermist positions, but they are also what >90% of people in every nation on earth already believe.

Reply
Load More