i currently believe that working on superintelligence-alignment is likely the correct choice from a fully-negative-utilitarian perspective.^[1]

for others, this may be an intuitive statement or unquestioned premise. for me it is not, and i'd like to state my reasons for believing it, partially as a response to this post concerned about negative utilitarians trying to accelerate progress towards an unaligned-ai-takeover.

there was a period during which i was more uncertain about this question, and avoided openly sharing minimally-dual-use alignment research (but did not try to accelerate progress towards a nonaligned-takeover) while resolving that uncertainty.

a few relevant updates since then:

1. decrease on the probability that the values an aligned AI would have would endorse human-caused moral catastrophes such as human-caused animal suffering.
   
   i did not automatically believe humans to be good-by-default, and wanted to take time to seriously consider what i think should be a default hypothesis-for-consideration upon existing in a society that generally accepts an ongoing mass torture event.
2. awareness of vastly worse possible s-risks.
   
   factory farming is a form of physical torture, by which i mean torture of a mind which is done through the indirect route of effecting its input channels (body/senses). it is also a form of psychological torture. it is very bad, but situations which are magnitudes worse seem possible, where a mind is modulated directly (on the neuronal level) and fully.
   
   compared to 'in-distribution suffering' (eg animal suffering, human-social conflicts), i find it further less probable that an AI aligned to some human-specified values^[2] would create a future with this.
   
   i think it's plausible that it exists rarely in other parts of the world, though, and if so would be important to prevent through acausal trade if we can.
3. (also see Kaj Sotala's reply about some plausible incidental s-risks)

i am not free of uncertainty about the topic, though.

in particular, if disvalue of suffering is common across the world, such that the suffering which can be reduced through acausal trade will be reduced through acausal trade regardless of whether we create an AI which disvalues suffering, then it would no longer be the case that working on alignment is the best decision for a purely negative utilitarian.

despite this uncertainty, my current belief is that the possibility of reducing suffering via acausal trade (including possibly such really-extreme forms of suffering) outweighs the probability and magnitude of human-aligned-AI-caused suffering.^[3]

also, to be clear, if it ever seems that an actualized s-risk takeover event is significantly more probable than it seems now^[4] as a result of unknown future developments, i would fully endorse causing a sooner unaligned-but-not-suffering takeover to prevent it.

1. ^{^}

   i find it easier to write this post as explaining my position as "even for a pure negative utilitarian, i think it's the correct choice", because it lets us ignore individual differences in how much moral weight is assigned to suffering relative to everything else.

   i think it's pretty improbable that i would, on 'idealized reflection'/CEV, endorse total-negative-utilitarianism (which has been classically pointed out as implying, e.g, preferring a universe with nothing to a universe containing a robust utopia plus an instance of light suffering).

   i self-describe as a "suffering-focused altruist" or "negative-leaning-utilitarian." ie, suffering seems much worse to me than happiness seems good.

2. ^{^}

   (though certainly there are some individual current humans would do this, for example to digital minds, if given the ability to do so. rather, i'm expressing a belief that it's very probable that an aligned AI which practically results from this situation would not allow that to happen.)

3. ^{^}

   (by 'human-aligned AI', I mean one pointed to an actual CEV of one or a group of humans (which could indirectly imply the 'CEV of everyone' but without actually-not-being-that and failing in the below way, and without allowing cruel values of some individuals to enter into it).

   I don't mean an AI aligned to some sort of 'current institutional process', like voting, involving all living humans -- I think that should be avoided due to politicization risk and potential for present/unreflective(/by which i mean cruel)-values lock-in.)

4. ^{^}

   there's some way to formalize with bayes equations how likely, from a negative-utilitarian perspective, an s-risk needs to be (relative to a good outcome) to terminate a timeline.

   it would intake probability distributions related to 'the frequency of suffering-disvalue across the universal distribution of ASIs' and 'the frequency of various forms of s-risks that are preventable with acausal trade'. i might create this formalization later.

   if we think there's pretty certainly more preventable-through-trade-type suffering-events than there is altruistic ASIs to prevent it, a local preventable-type s-risk might actually need to be 'more likely than the good/suffering-disvaluing outcome'