dinfinity

I would argue that more focus is warranted on values as being emergent, independent of whether they are in the training data in some form. The right to live for instance feels very fundamental morally, but it can also be seen as a necessary means for effective cooperation amongst individuals and thus something that will emerge in sufficiently advanced societies of a certain type. The term to put this under would be moral convergent evolution.

It's not a given that in ASI such a value would emerge as it would be pretty dissimilar to a human. Nevertheless I t... (read more)

Nothing is 'proven' with respect to future systems; one merely presents arguments, and this post is a series of arguments toward the conclusion that alignment is a real, unsolved problem that does not go well by default.

Do you find the claim "ASI is very likely to pursue the wrong goals" particularly well supported by the arguments made in that section of the article? I personally see mainly arguments why we can't make it pursue our goals (which I agree with), but that is not the same thing as showing that ASI is unlikely to land on 'good' goals (for hu... (read more)

I think that is the weakest point of this post and I would say this is an unsupported claim: "ASI is very likely to pursue the wrong goals."

Even if we do not manage to actively align ASI with our values and goals (which I do see pretty well argued in the post), it is unproven that it it is unlikely that ASI will not self-align or (in its self-optimization process) develop values that are benevolent towards us. Mass enslavement and/or actively working towards extinction of humanity are pretty high-friction and potentially risky paths. Cooperation, appeaseme... (read more)

cards

Typo: cards instead of cars.

I think we are in agreement that the consciousness is tied to the brain. Claiming equivalency is not warranted, though: The brain of a dead person (very probably, I'm sure you'd agree) contains no consciousness. Let's not dwell on this, though: I am definitely not claiming that consciousness exists outside of the brain, just that asserting physical continuity of the brain is not enough by itself to show continuity of conscious experience.

With regard to the modifications: Your line of reasoning runs into the classic issues of philosophical identity, as show... (read more)

I agree on the physical continuity of the brain, but I don't think this transfers to continuity of the consciousness or its experience. It is defining "you" as that physical brain, rather than the conscious experience itself. It's like saying that two waves are the same because they are produced by the same body of water.

Imagine significant modifications to your brain while you are asleep in such a way that your memories are vastly different, so much as to represent another person. Would the consciousness that is created on waking up experience a connectio... (read more)

I would say that it is irrelevant for the points the post/Rob is trying to make whether consciousness is classical or quantum, given that conscious experience has, AFAIK, never been reported to be 'quantum' (i.e. that we don't seem to experience superpositions or entanglement) and that we already have straightforward classical examples of lack of conscious continuity (namely: sleeping).

In the case of sleeping and waking up it is already clear that the currently awake consciousness is modeling its relation to past consciousnesses in that body through memori... (read more)