I'm now going to answer a slightly different question, which "Why is discussion of this sort of downvoted and dismissed sometimes?"

There is a vibe that I often get from suffering focused people, which is a combo of

a) seeming to be actively stuck in some kind of anxiety loop, preoccupied with hell in a way that seems more pathological to me than well-reasoned. 

b) something about their writing and vibe feels generally off, 

c) negative-utilitarians seem very frequently to me to be highly depressed, and I think the sort of person who ends up highly suffering focused rather than incorporating positive experiences into their agenda tend to be living in a world where they literally can't experience pleasure/good-things. 

I don't think any of this is necessary to care about s-risks (or even to be negative utilitarian). But I think it is common enough that a) sometimes people are downvoting/dismissing this because they're picking up correctly on this vibe, b) sometimes people are just... anticipating that vibe, maybe seeing it where it wasn't necessarily.

(oddly, I get this more from S-risk people than from Animal Rights people. Animal Rights people seem more motivated from 'man, atrocities are happening right now, and we should care about these for the same reason we generally care about atrocities. S-Risk people seem more often like they're trapped in a cognitive loop imagining the worst hell they can dream up no matter how useful that is)

Socially emergent avoidance. This is a secondary effect (and could explain anything if taken too strongly). My intuition is that there's a freaked-out-ness around S-risks that goes deeper than mere X-risk. This freaked-out-ness might be about the things you mention about NNTs or NU-affiliation, and also "sad". But it looks on the surface, to others, like general freaked-out-ness. When people around you are freaked out by something, you avoid talking about it. That snowballs, where people around you are avoiding talking about something, so you assume you're supposed to avoid talking about it.

i'll say that while i'm absolutely horrified at the possibility of S-risks, i think they're somewhat small, and that the work i'm doing now (fairly S-risk-resistant alignment) is pretty convergent to both S-risk and X-risk reduction.

in particular, an aligned AI sells more of its lightcone to get baby-eating aliens to eat their babies less, and in general a properly aligned AI will try its hardest to ensure what we care about (including reducing suffering) is satisfied, so alignment is convergent to both.

but some wonkier approaches could be pretty scary.

(note that another reason i don't think about S-risks too much is that i don't think my mental health could handle worrying about them a lot, and i need all the mental health i can get to solve alignment.)

I don't believe that reducing s-risks from AI involves substantially different things than those you'd need to deal with AI alignment.

Solving alignment solves s-risk as well as x-risks.

The main solutions to s-risks that don't work for x-risks are to facilitate extinction so that there's nothing left to suffer.

There are huge cost to advocating for human extinction to prevent s-risk.

Too unlikely. I’ve heard three versions of this concern. One is that s-risks are unlikely. I simply don’t think it is as explained above, in the post proper. The second version is that it’s 1/10th of extinction, hence less likely, hence not a priority. The third version of this take is that it’s just psychologically hard to be motivated for something that is not the mode of the probability distribution of how the future will turn out (given such clusters as s-risks, extinction, and business as usual). So even if s-risks are much worse and only slightly less likely than extinction, they’re still hard for people to work on.

NNTs. Some might argue that “naive negative utilitarians that take ideas seriously” (NNTs) want to destroy the world, so that any admissions that s-risks are morally important in expectation should happen only behind closed doors and only among trusted parties.

My take is that x-risk from misalignment is far more likely than s-risk, and that work to reduce loss of control of humanity to AGI is (for now) a sufficiently general pursuit to cover both cases. Maybe at some point there will be enough researchers to specialize more.

In the language of Superintelligent AI is necessary for an amazing future but far from sufficient, I expect that the majority of possible s-risks are weak dystopias rather than strong dystopias. We're unlikely to succeed at alignment enough and then signflip it (like, I expect strong dystopia to be dominated by 'we succeed at alignment to an extreme degree' ^ 'our architecture is not resistant to signflips' ^ 'somehow the sign flips'). So, I think literal worse-case Hell and the immediate surrounding possibilities are negligible.
I expect that the extrema of most AIs, even ones with attempted alignment patches, to be weird and unlikely to be of particular value to us. The ways values resolve has a lot of room to maneuver early on, before it becomes a coherent agent, and I don't expect those to have extrema that are best fit by humans (see various of So8res other posts). Thus, I think it is unlikely that we end up with a weak dystopia (at least for a long time, which is the s-risk) relative to x-risk.

My comments on this topic have been poorly received. I think most people are pretty much immune to the emotional impact of AI hell as long as it isn't affecting someone in their 'monkeysphere' (community of relationships capped by Dunbar's number).

The popular LW answer seems to be the top comment from Robin Hanson to my post here: https://www.lesswrong.com/posts/BSo7PLHQhLWbobvet/unethical-human-behavior-incentivised-by-existence-of-agi

My other more recent comment: https://www.lesswrong.com/posts/pLLeGA7aGaJpgCkof/?commentId=rWePAitP2syueDf25

Arguably, if you're concerned about s-risk, you should be theorizing about ways of controlling access to Em data. You would be interested in better digital rights management (DRM) technology, which is seen as 'the enemy' in a lot of tech/open-source adjacent communities, as well as developing technology for guaranteed secure deletion of human consciousness.

If it were possible to emulate a human and place them into AI hell, I am absolutely certain that the US government would find a way to use it for both interrogation and incarceration.

Speaking for myself: I care a bunch about s-risks. I listed "the AI might simulate sentient beings" as a failure mode in "Carefully Bootstrapped Alignment" is organizationally hard.

But I overall think working on alignment is largely more urgent. Being able to understand what's going on at all inside a neural net, and advocating that companies be required to understand what's going on before developing new/bigger/better models, seems like a convergent goal relevant to both human extinction and astronomical suffering. 

There's also something of a Maslow Hierarchy of needs thing where I think getting to "okay, we're not all dead and we have some control over the future" seems more like the next major step to focus on, in part since it requires fewer philosophical assumptions.

But I am pretty in favor of people a) figuring out how to build AI evals that check for sentience, and/or finding a reasonable bright line that's like "if your AI can do X, there's at least a sizable chance that it's sentient", and b) advocating companies or governments halt training runs or deployment if that bright line is crossed. 

Note, like, Eliezer did go out of his way to mention AI consciousness as a problem in his TIME article and in some tweets. 

As I'd mentioned elsewhere,

The one good thing about the [nature of the] technical problem of alignment is that it makes hyperexistential risks — the risks of astronomical suffering — very unlikely.

The problem of AI Alignment can be viewed as the problem of encoding our preferences into an AGI, bit by bit. The strength of alignment tools, in turn, translates to how many bits we can encode. With the current methods of end-to-end training, we're essentially sampling preferences at random. Perfect interpretability and parameter-surgery tools would allow us to encode an arbitrary amount of bits. The tools we'll actually have will be somewhere between these two extremes.

"Build us our perfect world" is a very complicated ask, and it surely takes up many, many thousands of bits. That's why alignment is hard.

"Build us a hell" is its mirror. It's essentially the same ask, except for a flipped sign. As such, specifying it would require pretty much the same amount of bits.

Thus, in the timelines where we have alignment tools advanced enough to build a hell-making AGI, it's overwhelmingly likely that we have the technical capability to build an utopia-building AGI. On the flipside, conditioning on our inability to build an utopia-builder, our tools are probably so bad we can't come close to a hell-builder. In that case, we just sample some random preferences, and the AGI kills us quickly and painlessly.

Screwing up so badly we create a suffering-maximizer is vanishingly unlikely: it's only possible in a very, very narrow range of technical capabilities.

I am worried about S-risks though. I think they're pretty likely in timelines where we solve the technical problem of alignment, but the technology ends up in the wrong hands; central examples being xenophobic or authoritarian political entities.

I'm concerned it may be neglected, too: I expect the various AI Governance/field-building initiatives may not be spending any time considering how not to attract the wrong kind of attention, and instead simply maximize for getting as much attention as possible. (Though I suppose if they're competent at it, I wouldn't see any public evidence of them considering that; I'm just guessing on priors.)

Edit: Mm, though there's a caveat. I'm operating under the least forgiving model of the alignment problem; under it, S-risks really are that unlikely. But many people don't share it — e. g., the shard theory assumes "rough" alignment will suffice to avoid omnicide — which should make their P(hell) non-negligibly high. Yet they're not worried either, so there must be something else going on with their models.

S-risks aren't disproportionately important to many people. Personally I think they're only a bit worse than death, and aesthetically I think they're maybe twice as bad as human extinction.

This should be combined with a likelihood estimate to recommend actions.

One of the reasons why I'm skeptical of the S-Risk is as follows. 

Not sure if it's a core idea, but I've observed that S-Risk proponents often propagate the idea that some large amount of suffering is worse than death. 

For example, some of them claim that assisted suicide for a patient in pain is ethical (the claim which I find abhorrent, unless the procedure is done for cryonics).

My view is, there NO fate worse than death.  A single human death is worse than trillions of years of the worst possible suffering by trillions of people. 

The "some suffering is worse than death" idea is increasing X-risks: one day, some sufficiently powerful idiot could decide that human extinction is better than an AGI-dystopia. 

It's a good idea to work on preventing large amounts of suffering, but S-Risk is a bad framework for that. 

I think there's an additional freaked-out-ness that comes from a thing related to NNTs. But less about being worried that people will try to destroy the world. Instead, more like not wanting to inhabit or cause other people to inhabit a minset where self-destruction would be an obvious choice. Suicide is a fairly taboo topic. And my impression is that this is at least a little justified; my pop-sci level understanding is that suicide being raised to attention (especially by a celebrity killing themselves) can cause other people to be more likely to kill themselves.

That could simply be from raising the hypothesis to attention.

There could be an intuitive fear of self-fulfilling prophecies. If you start to believe that it would be better if you didn't exist, one reaction might be to stop making your life better to live--which would at least somewhat contribute toward it becoming true that it'd be better if you didn't exist. (Which could be in a self-reinforcing loop.)

Too unknown. Finally there’s the obvious reason that people just don’t know enough about s-risks. That seems quite likely to me.

The biggest extinction risk from AI comes from instrumental convergence for resource acquisition in which an AI not aligned with human values uses the atoms in our bodies for whatever goals it has.  An advantage of such instrumental convergence is that it would prevent an AI from bothering to impose suffering on us.

Unfortunately, this means that making progress on the instrumental convergence problem increases S-risks.  We get hell if we solve instrumental convergence, but not, say, mesa-optimization and we get a powerful AGI that cares about our fate, but does something to us we consider worse than death.

Some reasons why Im personally not as involved in working to prevent AI Hell:

(in no order of importance).

1. Im not strongly convinced a hostile Singularity is plausible at least in the near future, from technological, logistical, and practical standpoint. Pretty much every AI Hell scenario I have read, hinges on sudden appearance of scientifically implausible technologies, and on instant perfect logistics that the AI could use.

2. Main issue that could lead to AI Hell is the misalignment of values between AI and humans. However, it is patently obvious that humans are not aligned with each other, with themselves or with rational logic. Therefore, I do not see a path to align AI with human values unless we Solve Ethics, which is an impossible task unless we completely redesign human brains from scratch.

3. Im personally not qualified to work on any technological aspects of preventing AI Hell. I am qualified to work on human-end Ethics and branch into alignment from that, and I see it as an impossible task with the kind of humans we get to work with.

4. A combination of points 1 and 2 leads me to believe that humanity is far more likely to abuse early stage AI to wipe itself out, than for AI itself to wipe out humanity of its own volition. To put it differently, crude sub-human level AI can plausibly be used to cause WW3 and a nuclear holocaust without any need for hostile superhuman AI. I think we worry too much about the unlikely but extremely lethal post-Singularity AI, and not enough about highly likely and just sufficiently lethal wargame systems in the hands of actual biological humans, who are not sufficiently concerned with humanity's survival.

5. Roko's Gremlin: anyone who is actively working on limiting or forcibly aligning AI is automatically on the hit-list of any sufficiently advanced hostile AI. Im not talking about long term high-end scenario of the Roko's Basilisk, but rather the near-future low-end situation in which an Internet savvy AI can ruin your life for being a potential threat to it. In fact, this scenario does not require actively hostile AI at all. I see it as completely plausible that a human being with a vested financial interest in AI advancement could plausibly use AI to create a powerful smear campaign against, say, EY, to destroy his credibility, and with him the credibility of the AI Safety movement. Currently accessible AI is excellent at creating plausible-seeming bullshit, which would be perfect to use for social media warfare against anyone who tries to monkeywrench its progression. Look at Nick Bostrom to see how easily one of us can be sniped down with minimum effort.

In addition to many good points already mentioned, I would like to add that I have no idea how to approach this problem.

Approaching x-risk is very hard too, but it is much clearer in comparison.

Personal fit. Surely, some people have tried working on s-risks in different roles for some substantial period of time but haven’t found an angle from which they can contribute given their particular skills.

Egoism, presentism, or substratism. The worst s-risks will probably not befall us (humans presently alive) or biological beings at all. Extinction, if it happens, will. Maybe death or the promise of utopia has a stronger intuitive appeal to people if they themselves have a risk/chance of experiencing it?

NUs. Some people may think that you have to be a negative utilitarian to care about s-risks. They are not negative utilitarians, so they steer clear of the topic.

Too unpopular. Maybe people are motivated by what topics are in vogue in their friend circles, and s-risks are not?

Too sad. Some people think that maybe working on s-risks is unpopular because suffering is too emotionally draining to think about, so people prefer to ignore it.

Another version of this concern is that sad topics are not in vogue with the rich tech founders who bankroll our think tanks; that they’re selected to be the sort of people who are excited about incredible moonshots rather than prudent risk management. If these people hear about averting suffering, reducing risks, etc. too often from EA circles, they’ll become uninterested in EA-aligned thinking and think tanks.

We don't even care about actually existing hells. We just make the social signals that appear like caring insofar as they're expected of us, method-acting the emotions for exactly as long as is appropriate. Hence, we cry "that's horrible!" when we hear that a hundred million chickens literally suffer to death in factory farms annually, only to stop thinking about it ASAP so that we can keep making fun of vegans and eating chicken nuggets without being overwhelmed by the terror and nausea and rage. Hence, we look down when we see pictures of starving children, unwilling to bear the pain of empathy, but dissimulate about "local responsibility" or "warlords" whenever the idea of using money to alleviate the problem comes up. Hence, we feel a wretched blackness envelop our hearts when a nature documentary shows us a wide-eyed deer screaming as a cougar rips its unborn calf out of its stomach, only to about-face at the very moment that someone raises the question of doing anything about it, smugly telling them that it's not in our moral jurisdiction. 

Unlike AI safety in general, the study of s-risks isn't particularly large, technical, or drama-filled, so the typical person who ends up exposed to the idea—a shape rotator with something to prove—tends not to be drawn to it in particular. Those that are attracted to it will generally have been attracted because it was introduced to them in some particularly spicy context, or because it finds a perfect fit in their pre-existing structure of neuroses; they will say that they're working on it out of a sense of ethical prioritization, and they'll believe themselves, but somehow it's always their personal fascinations and status struggles that end up being prioritized. If you want to get more people working on s-risks, then, you might want to nerd-snipe them with math-y dilemmas, like why suffering necessarily takes place over time, and the questions this poses for ethical optimization at relativistic scales, or, what protocols an agent might deploy for detecting and undoing adversarial modifications of their utility function like sign-flipping (keeping in mind that such a protocol must act automatically, since the agent won't want to trigger it). 

Also, a nitpick, or maybe a miscommunication, but: Death with Dignity is not about giving up the fight: it's about dropping the unrealistic expectation that we'll somehow end up making it, while still continuing to, in Eliezer's own words, "increase the log odds of Earth's survival".