The idea of an "aligned superintelligence" seems misguided

[-]JBlack3y40

The phrase "aligned superintelligence" is pretty broad. At one end, is the concept of a superintelligence that will do everything we want and nothing we don't, for some vague inclusive sense of "what we want". At the other end is superintelligence that won't literally kill everyone as a byproduct of doing whatever it does.

Obviously the first requires the second, but we don't know how to ensure even the second. There is an unacceptably high risk that we will get superintelligence that does kill everyone, though different people have quite different magnitudes for exactly how unacceptably high risk it is.

The biggest issue is that if the superintelligence "wants" anything (in the sense of carrying out plans to achieve it), we won't be able to prevent those. So a large part of alignment is either ensuring that it never actually "wants" anything (e.g. tool AI), or that we can somehow develop it in a way that it "wants" things that are compatible with our continued existence and ideally, flourishing. This is the core idea of "alignment".

If alignment is misguided as a whole, then either superintelligence never occurs or humanity ends.

[-]ssadler3y10

So the first one is an "AGSI", and the second is an "ANSI" (general vs narrow)?

If I understand correctly... One type of alignment (required for the "AGSI") is what I'm referring to as alignment which is that it is conscious of all of our interests and tries to respect them, like a good friend, and the other is that it's narrow enough in scope that it literally just does that one thing, way better than humans could, but the scope is narrow enough that we can hopefully reason about it and have an idea that it's safe.

Alignment is kind of a confusing term if applied to ANSI, because to me at least it seems to suggest agency and aligned interests, wheras in the case of ANSI if I understand correctly the idea is to prevent it from having agency and interests in the first place. So it's "aligned" in the same way that a car is aligned, ie it doesn't veer off the road at 80 mph :-)

But I'm not sure if I've understood correctly, thanks for your help...

[-]M. Y. Zuo3y20

This interesting post was sitting at -1 total karma with 5 votes at the time of reading, so I strongly upvoted. It contains a brief but cogent argument that seems novel on LW. Though it might have been made before in more abstract and wordy terms, I still think the novelty of brevity is valuable.

[-]ssadler3y10

Thank you kind fellow !

[-]Joe Collman3y20

I think your intuitions are essentially correct - in particular, I think many people draw poor conclusions from the strange notion that humans are an example of alignment success.

However, you seem to be conflating:

Fully general SI alignment is unsolvable (I think most researchers would agree with this)
There is no case where SI alignment is solvable (I think almost everyone disagrees with this)

We don't need to solve the problem in full generality - we only need to find one setup that works.
Importantly, 'works' here means that we have sufficient understanding of the system to guarantee some alignment property with high probability. We don't need complete understanding.

This is still a very high bar - we need to guarantee-with-high-probability some property that actually leads to things turning out well, not simply one which satisfies my definition of alignment.

I haven't seen any existence proof for a predictably fairly safe path to an aligned SI (or aligned AGI...).
It'd be nice to know that such a path existed.

[-]ssadler3y10

Thanks for your response! Could you explain what you mean by "fully general"? Do you mean that alignment of narrow SI is possible? Or that partial alignment of general SI is good enough in some circumstance? If it's the latter could you give an example?

[-]Joe Collman3y10

By "fully general" I mean something like "With alignment process x, we could take the specification of any SI, apply x to it, and have an aligned version of that SI specification". (I assume almost everyone thinks this isn't achievable)

But we don't need an approach that's this strong: we don't need to be able to align all, most, or even a small fraction of SIs. One is enough - and in principle we could build in many highly specific constraints by construction (given sufficient understanding).

This still seems very hard, but I don't think there's any straightforward argument for its impossibility/intractability. Most such arguments only work against the more general solutions - i.e. if we needed to be able to align any SI specification.

Here's a survey of a bunch of impossibility results if you're interested.
These also apply to stronger results than we need (which is nice!).

LESSWRONG
LW

LESSWRONG
LW

6

The idea of an "aligned superintelligence" seems misguided

6

6