It seems likely to me that "driving a car" used as a core example actually took something like billions of person-years including tens of millions of fatalities to get to the stage that it is today.

Some specific human, after being raised for more than a dozen years in a social background that includes frequent exposure to car-driving behaviour both in person and in media, is already somewhat primed to learn how to safely drive a vehicle designed for human use within a system of road rules and infrastructure customized over more than a century to human skills, sensory modalities, and background culture. All the vehicles, roads, markings, signs, and rules have been designed and redesigned so that humans aren't as terrible at learning how to navigate them as they were in the first few decades.

Many early operators of motor vehicles (adult, experienced humans) frequently did things that were frankly insane by modern standards and would send an AI driving research division back to redesign if their software did such things even once.

It's definitely not a coherent logic as those are defined to be first-order, while this is explicitly a second-order logic.

One large difference between the scenarios is the answer to "what's in it for the stranger?"

In the standard Pascal's Mugging, the answer is "they get $5". Clear and understandable motivation, it is after all a mugging (or at least a begging). It may or may not escalate to violence or other unpleasantness if you refuse to give them $5, though there's a decent chance that if you do give them $5 then they'll bug you and other people again and again.

In this scenario it's much less clear. What they're saying is obviously false, but they don't obviously get much that would incentivize bugging you again if you pretend to agree with them. It's a bit disconcerting that they've been tracking your movements every day, but it's not like you particularly lose anything by not eating an apple that they've warned you against eating and you weren't planning to eat anyway. Frankly, even without them claiming to own the apples, I'd be pretty disinclined to eat one that this stalker has pointed out to me.

Not at all. You may be able to see a positional advantage or capture of a minor piece in your move, and not see that they can respond by capturing your queen. The most apparently valuable moves after your own move are very often close to the worst after theirs, because they are often made with the most powerful pieces and expose them to risk.

I learned that lesson quite well when writing my own poor attempt at a chess playing program years ago. Odd ply searches are generally worse than even ones for this reason.

a) You have to decide on a moral framework that can be explained in detail, to anyone.

b) It will be implemented worldwide tomorrow. 

c) Tomorrow, every single human on Earth, including you and everyone you know, will also have their lives randomly swapped with someone else.

This is a fridge horror scenario. The more I consider it, the creepier and more horrifying it gets.

So 8 billion people are going to find themselves in someone else's body of random age and sex, a large fraction unable to speak with the complete strangers around them, with no idea what has happened to their family, their existing moral framework ripped out of their minds and replaced with some external one that doesn't fit with their preexisting experiences, with society completely dysfunctional and only one person who knows what's going on?

Regardless of details of whatever moral framework is chosen, that's an immense evil right there. If fewer than a billion people die before food infrastructure is restored, it would be a miracle.

Unlike ASI, some forms of biological superintelligence already exist and have for a long time, and we call them corporations, nation states, and other human organizations.

Most of these social structures are, in the aggregate, substantially stupider than individual humans in many important ways.

This is pretty close to the dust theory of Greg Egan's Permutation City and also similar in most ways to Tegmark's universe ensemble.

They are the only options available in the problem. It is true that this means that some optimality and convergence results in decision theory are not available.

It's not exotic at all. It's just a compatibilist interpretation of the term "free will", which form a pretty major class of positions on the subject.

That doesn't address the question at all. That just says if the system is well modelled as having a utility function, then ... etc. Why should we have such high credence that the premise is true?