This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I was going to say that this seems like another one of Eliezer's founder effects, but he actually wrote about not trusting humans with too much power, in Creating Friendly AI 1.0 (2001):
Among humans, the only practical way to maximize actual freedom (the percentage of actions executed without interference) is to ensure that no human entity has the ability to interfere with you—a consequence of humans having an innate, evolved tendency to abuse power. Thus, a lot of our ethical guidelines (especially the ones we’ve come up with in the twentieth century) state that it’s wrong to acquire too much power.
If this is one of those things that simply doesn’t apply in the spaces beyond the Singularity—if, having no evolved tendency to abuse power, no injunction against the accumulation of power is necessary—one of the possible resolutions of the Singularity would be the Sysop Scenario. The initial seed-AI-turned-Friendly-superintelligence, the Transition Guide, would create (or self-modify into) a superintelligence that would act as the underlying operating system for all the matter in human space—a Sysop.
I wonder how to interpret e.g. the ending of HPMOR in light of this.