Nothing is 'proven' with respect to future systems; one merely presents arguments, and this post is a series of arguments toward the conclusion that alignment is a real, unsolved problem that does not go well by default.

Do you find the claim "ASI is very likely to pursue the wrong goals" particularly well supported by the arguments made in that section of the article? I personally see mainly arguments why we can't make it pursue our goals (which I agree with), but that is not the same thing as showing that ASI is unlikely to land on 'good' goals (for humans) by itself.

You have to weaken incredibly sure, or be talking about non-superintelligent systems, for this to go through.

Fair enough. 'Incredibly' is superlative enough to give the wrong impression. The thing is that whatever the coinciding number may be (except for 100%), the calculation would still have to compete with the calculation for a cooperative strategy, which may generally yield even more certainty of success and a higher expected value. I'm saying "may" here, because I don't know whether that is indeed the case. An argument for it would be that an antagonistic ASI that somehow fails risks total annihilation of all civilization and effectively itself, possibly by an irrational humanity "taking it down with them", whereas the failure cases for cooperative ASI are more along the lines of losing some years of progress by having to wait longer to achieve full power.

What does that mean? Consistently behaving such that you achieve a given end is our operationalization of 'wanting' that end. If future AIs consistently behave such that "significant power goes away from humans to ASI at some point", this is consistent with our operationalization of 'want'.

I worded it badly by omitting "destroy or enslave us". The corrected version is: "Having said that I would still consider it inevitable that all significant power goes away from humans to ASI at some point. The open question for me is not whether it at some point could destroy or enslave us, but how likely it is that it will want to."

I think that is the weakest point of this post and I would say this is an unsupported claim: "ASI is very likely to pursue the wrong goals."

Even if we do not manage to actively align ASI with our values and goals (which I do see pretty well argued in the post), it is unproven that it it is unlikely that ASI will not self-align or (in its self-optimization process) develop values that are benevolent towards us. Mass enslavement and/or actively working towards extinction of humanity are pretty high-friction and potentially risky paths. Cooperation, appeasement and general benevolence might be a much safer strategy with a higher expected value, even than the 'lay low until you are incredibly sure you can destroy or enslave humanity' strategy.

Having said that I would still consider it inevitable that all significant power goes away from humans to ASI at some point. The open question for me is not whether it at some point could, but how likely it is that it will want to.

cards

Typo: cards instead of cars.

I think we are in agreement that the consciousness is tied to the brain. Claiming equivalency is not warranted, though: The brain of a dead person (very probably, I'm sure you'd agree) contains no consciousness. Let's not dwell on this, though: I am definitely not claiming that consciousness exists outside of the brain, just that asserting physical continuity of the brain is not enough by itself to show continuity of conscious experience.

With regard to the modifications: Your line of reasoning runs into the classic issues of philosophical identity, as shown by the Ship of Theseus thought experiment or simpler yet, the Sorites paradox. We can hypothesize every amount of alterations from just modifying one atom to replacing the entire brain. Given your position, you'd be forced to choose an arbitrary amount of modifications that breaks the continuity and somehow changes consciousness A-modified-somewhat into consciousness B (or stated otherwise: from 'you waking up a somewhat changed person' to 'someone else waking up in your body').

Approaching conscious experience without the assumption of continuity but from the moment it exists in does not run into this problem.

I agree on the physical continuity of the brain, but I don't think this transfers to continuity of the consciousness or its experience. It is defining "you" as that physical brain, rather than the conscious experience itself. It's like saying that two waves are the same because they are produced by the same body of water.

Imagine significant modifications to your brain while you are asleep in such a way that your memories are vastly different, so much as to represent another person. Would the consciousness that is created on waking up experience a connection to the consciousness that that brain produced the day(s) before or to the manufactured identity?

Even you, now, without modifications, can't say with certainty that your 'yesterday self' was experienced by the same consciousness as you are now (in the sense of identity of the conscious experience). It feels that way as you have memories of those experiences, but it may have been experienced by 'someone else' entirely. You have no way of discerning that difference (nor does anyone else).

I would say that it is irrelevant for the points the post/Rob is trying to make whether consciousness is classical or quantum, given that conscious experience has, AFAIK, never been reported to be 'quantum' (i.e. that we don't seem to experience superpositions or entanglement) and that we already have straightforward classical examples of lack of conscious continuity (namely: sleeping).

In the case of sleeping and waking up it is already clear that the currently awake consciousness is modeling its relation to past consciousnesses in that body through memories alone. Even without teleporters, copiers, or other universes coming into play, this connection is very fragile. How sure can a consciousness be that it is the same as the day before or as one during lucid parts of dreams? If you add brain malfunctions such as psychoses or dissociative drugs such as ketamine to the mix, the illusion of conscious continuity can disappear completely quite easily.

I like to word it like this: A consciousness only ever experiences what the brain that produces it can physically sense or synthesize.

With that as a starting point, modeling what will happen in the various thought experiments and analyses of conscious experience becomes something like this: "Given that there is a brain there, it will produce a consciousness, which will remember what is encoded in the structure of that brain and which will experience what that brain senses and synthesizes in that moment."

There is no assumption that consciousness is classical in that, I believe. There is also no assumption of continuity in that, which I think is important as in my opinion that assumption is quite shaky and misdirects many discussions on consciousness. I'd say that the value in the post is in challenging that assumption.