I didn't mean it to be so simplistic. I am just considering that if there is a known limitation of AI, no matter how powerful it is, that could be used as the basis of a system an AI could not circumvent. For example, if there was a shutdown system where the only way to disable it would require solving the halting problem.

While this is an example of international cooperation in the face of mutually-assured destruction, there is some historical context that shows why this is effective, in my opinion: First, because the destructive power of nuclear weapons was already realized by Hiroshima and Nagasaki. Second, because the US no longer had a monopoly on nuclear weapons and determined that overcoming the USSR by force was no longer an option. Third, because the world was still recovering from the Second World War and shared a universal desire for peaceful resolutions.

The implication that I didn't think to spell out is that the AI should be programmed with the capacity for empathy. It's more of a proposal of system design than a proposal of governance. Granted, the specifics of that design would be its own discussion entirely

You were the chosen one, Anakin!

I think the harsh truth is that no one cared about Nuclear Weapons until Hiroshima was bombed. The concept of one nation "disarming" AI would never be appreciated until somebody gets burned.

I wonder if you could expand more on this observation. So you are saying that a dream is operating on a very limited dataset on a person, not an exact copy of information ("full description"). Do I understand right?

I sort of do intend of it as a kind of reductio, unless people find reason for this "Dream Hypothesis" to be taken seriously.

I don't see anything in that scenario that prevents a human-level AGI from using a collection of superintelligent tool AIs with a better interface to achieve feats of intelligence that humans cannot, even with the same tool AIs.

At that point, it wouldn't functionally be different than a series of tool AIs being controlled directly by a human operator. If that poses risk, then mitigations could be extrapolated to the combined-system scenario.

What fundamental law of the universe would set a limit right there, out of all possible capacities across every possible form of computing substrate?

I'm not trying to imply there is something about the human mind specifically that forces a limit to computing power, I just used that as a benchmark as that is the only frame of reference that we have. If it is dumber or slightly smarter than a human on the same order of magnitude, that doesn't really matter.

The concept of a trade-off is simply saying that the more complex a system is to imitate consciousness, the more computational ability is sacrificed, tending towards some lower bound of computational substrate that one may not count as superintelligent. I'm not saying I have any physical or informational-theoretical law in mind for that currently, though.

Isn't a deceptive agent the hallmark of unfriendly AI? In what scenarios does a dishonest agent reflect a good design?

Of course, I didn't mean to say that TDT always keeps its promises, just that it is capable of doing so in scenarios like Parfit's Hitchhiker, where CDT is incapable of doing so.

Thanks, I'll be sure to check them out

C,C is second-best, you prefer D,C and Nash says D,D is all you should expect. C,C is definitely better than C,D or D,D, so in the special case of symmetrical decisions, it's winning. It bugs me as much as you that this part gets glossed over so often.

I see what you mean, it works as long as both sides have roughly similar behavior.

Counterfactual Mugging is a win to pay off, in a universe where that sort of thing happens. You really do want to be correctly predicted to pay off, and enjoy the $10K in those cases where the coin goes your way.

For me, this would make intuitive sense if there was something in the problem that implied that Omega does this on a regular basis, analogous to the Iterated Prisoner's Dilemma. But as long as the problem is worded as a one-shot, once-in-a-lifetime scenario, then it comes off like the $10,000 is purely fictitious.