dinfinity

I think we are in agreement that the consciousness is tied to the brain. Claiming equivalency is not warranted, though: The brain of a dead person (very probably, I'm sure you'd agree) contains no consciousness. Let's not dwell on this, though: I am definitely not claiming that consciousness exists outside of the brain, just that asserting physical continuity of the brain is not enough by itself to show continuity of conscious experience.

With regard to the modifications: Your line of reasoning runs into the classic issues of philosophical identity, as shown by the Ship of Theseus thought experiment or simpler yet, the Sorites paradox. We can hypothesize every amount of alterations from just... (read more)

I agree on the physical continuity of the brain, but I don't think this transfers to continuity of the consciousness or its experience. It is defining "you" as that physical brain, rather than the conscious experience itself. It's like saying that two waves are the same because they are produced by the same body of water.

Imagine significant modifications to your brain while you are asleep in such a way that your memories are vastly different, so much as to represent another person. Would the consciousness that is created on waking up experience a connection to the consciousness that that brain produced the day(s) before or to the manufactured identity?

Even you, now, without... (read more)

I would say that it is irrelevant for the points the post/Rob is trying to make whether consciousness is classical or quantum, given that conscious experience has, AFAIK, never been reported to be 'quantum' (i.e. that we don't seem to experience superpositions or entanglement) and that we already have straightforward classical examples of lack of conscious continuity (namely: sleeping).

In the case of sleeping and waking up it is already clear that the currently awake consciousness is modeling its relation to past consciousnesses in that body through memories alone. Even without teleporters, copiers, or other universes coming into play, this connection is very fragile. How sure can a consciousness be that it is... (read more)

Nothing is 'proven' with respect to future systems; one merely presents arguments, and this post is a series of arguments toward the conclusion that alignment is a real, unsolved problem that does not go well by default.

Do you find the claim "ASI is very likely to pursue the wrong goals" particularly well supported by the arguments made in that section of the article? I personally see mainly arguments why we can't make it pursue our goals (which I agree with), but that is not the same thing as showing that ASI is unlikely to land on 'good' goals (for humans) by itself.

You have to weaken incredibly sure, or be talking about non-superintelligent

I think that is the weakest point of this post and I would say this is an unsupported claim: "ASI is very likely to pursue the wrong goals."

Even if we do not manage to actively align ASI with our values and goals (which I do see pretty well argued in the post), it is unproven that it it is unlikely that ASI will not self-align or (in its self-optimization process) develop values that are benevolent towards us. Mass enslavement and/or actively working towards extinction of humanity are pretty high-friction and potentially risky paths. Cooperation, appeasement and general benevolence might be a much safer strategy with a higher expected value, even than the... (read more)